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PREFACE 


Management  Consulting  and  Research,  Inc.  (MCR)  provided 
support  to  the  Office  of  the  Assistant  Secretary  of  Defense 
(OASD)  for  Manpower,  Reserve  Affairs  and  Logistics  (MRA&L) 
under  contract  number  MDA903-82-C-0278  for  the  examination  of 
skill  training.  MCR  analyses  will  assist  in  the  evaluation  and 
support  of  Service  training  programs. 

This  technical  report  is  a  contract  deliverable  that 
documents  the  skill  training  analyses  conducted  for  each  task  of 
this  project.  The  report  is  provided  in  two  volumes: 

•  Volume  I,  "Skill  Training  Analysis:  The  Linkage  of 
Unit  Level  Skill  Training  and  Unit  Productivity, "  and 

•  Volume  II,  "Skill  Training  Analysis:  An  Examination 
of  the  Navy  Pipeline  Management  System. " 

We  would  like  to  acknowledge  the  continuing  guidance  and 
assistance  of  Mr.  Michael  J.  Kendall,  COTR,  of  the  Training  and 
Education  Directorate,  and  the  assistance  provided  by  other  mem¬ 
bers  of  the  OSD  staff  and  the  Military  Service  staffs. 
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The  purpose  of  this  study  was  to  analyze  Service  skill 
training  conducted  at  the  installation  level  and  to  analyze  its 
impact  on  unit  productivity.  MCR  also  conducted  a  special  pur¬ 
pose  task  which  was  to  examine  the  Navy  Pipeline  Management  Sys¬ 
tem  with  emphasis  on  a  specific  critical  skill.  The  four  tasks 
we  performed  are  listed  below: 

•  Task  1  —  an  examination  of  the  impact  of  a  Field 
Training  Detachment  (FTD)  on  Air  Force  operational 
unit  productivity; 

•  Task  2  —  an  examination  of  the  impact  of  installation 
level  training  using  simulators  on  F-16  unit  mainten¬ 
ance  productivity ; 

•  Task  3  --  an  examination  of  the  impact  of  installation 
level  training  on  Army  operational  unit  maintenance 
productivity;  and 

•  Task  4  —  an  examination  of  the  Navy  Pipeline  Management 
system. 

This  report,  which  documents  our  work,  is  divided  into  two 
volumes:  Volume  I,  "Skill  Training  Analysis:  The  Linkage  of 

Unit  Level  Skill  Training  and  Unit  Productivity,"  and  Volume  II, 
"An  Examination  of  the  Navy  Pipeline  Management  System." 

Volume  I  of  the  report  describes  the  two  tasks  on  Air  Force 
installation-level  skill  training  and  the  task  on  Army  instal¬ 
lation-level  skill  training  listed  above.  In  the  first  Air  Force 


task  (described  in  Section  II  of  Volume  I),  we  developed  two 
methods  for  linking  skill  training  to  maintenance  productivity. 
In  the  second  Air  Force  task  (described  in  Section  III  of  Volume 
I),  we  developed  three  techniques  for  examining  the  linkage  bet¬ 
ween  training  and  maintenance  productivity.  In  the  Army  task 
(described  in  Section  IV  of  Volume  I),  we  examined  Army  instal¬ 
lation  level  training.  Section  IV  of  Volume  I  also  contains  a 
description  of  the  Army  training  effectiveness  analysis  process. 

Volume  II  of  the  report  contains  an  examination  of  the  Navy 
training  pipeline  with  specific  examination  of  two  Navy  ratings: 
Aviation  Electronics  Technician  and  Electronics  Technician. 


B .  OBSERVATIONS 

The  following  observations  were  made  in  the  course  of  our 
examination  of  skill  training.  The  observations  are  grouped 
by  each  of  the  tasks  we  performed  during  the  study. 

•  TASK  Is  This  task  attempted  to  illustrate  a  verifi¬ 
able,  positive  relationship  between  Field  Training 
Detachment  ( FTD)  training  and  job  performance.  Two 
methods  were  devised  to  illustrate  this  relationship: 
a  Quality  Assurance  (QA)  methodology,  which  compared 
individual  performance  evaluations  in  a  statistical 
manner;  and  a  Work  Unit  Code/Trend  Analysis  methodology, 
which  compared  average  time  to  complete  a  like  task 
between  work  centers.  "Macro"  measures  of  performance, 
such  as  the  number  of  aircraft  hours  flown  and  the 
number  of  aborted  flights  due  to  mechanical  problems, 
were  not  considered  in  this  task.  Although  these  mea¬ 
sures  might  be  more  accurate  maintenance  performance 
measures,  they  are  not  directly  relatable  to  training 
and  could  not  be  used.  The  results  of  our  work  in  this 
task  were  not  conclusive  although  they  indicate  that 
FTD  training  had  some  impact  on  productivity.  The 
measures  chosen  did  not  capture  significant  differ¬ 
ences,  but  this  is  explained  by  several  biases  that 
exist  in  the  data  that  was  used.  This  observation 


is  not  to  say  that  FTD  training  is  ineffective  in  terms 
of  teaching  new  job  skills.  One  must  realize  that  per¬ 
sonnel  who  go  through  FTD  training  have  already  re¬ 
ceived  extensive  training  in  their  specialty  fields, 
and  this  additional  unit  training  is  a  refinement,  or 
"add-on"  to  their  broad  training  base. 

The  advantages  of  FTD  training  are  in  three  areas: 

capability  of  rapid  adjustment  to  local  require¬ 
ments, 

-  cost  savings  based  on  little  or  no  need  to  travel 
to  distant  schools  for  training,  and 

-  rapid  return  of  students  to  the  job. 

Unit  training  goals  can  be  met  through  FTD  schooling 
and  through  an  on-the-job  training  (OJT)  program.  OJT 
allows  work  to  continue  without  loss  of  students  and 
instructors  to  the  local  school,  but  FTD  training  helps 
to  get  a  person  on  the  job  at  a  particular  skill  level 
in  a  shorter  period  of  time.  This  fact,  combined  with 
modest  increases  in  productivity  (measured  either  in 
quantity  or  quality  of  work),  should  produce  higher 
levels  of  aircraft  availability  without  an  increase  in 
the  size  of  the  work  force.  Any  increase  in  aircraft 
availability  yields  readiness  improvements.  This  may 
be  the  most  important  benefit  of  FTD  training. 

In  the  course  of  our  research  on  the  QA  methodology, 
we  had  the  opportunity  to  examine  the  work  of  the  QA 
section  at  several  wings.  The  wing  commander,  in  his 
attempt  to  achieve  the  highest  number  of  mission  avail¬ 
able  aircraft,  has  a  valuable  tool  in  his  QA  section. 
This  group  of  highly  skilled  maintenance  personnel  of 
varied  skills  performs  an  important  function  in  its 
evaluation  of  individual  mechanics.  Our  analysis  in¬ 
cluded  a  sample  of  2,180  personnel  tested  by  the  QA 
sections  at  our  sample  wings;  702,  or  32%,  failed  to 
pass  the  QA  certification.  These  personnel  had  to  be 
retrained  and  then  recertified  by  their  supervisors 
that  they  were  capable  of  performing  their  work.  Thus, 
we  found  that  the  QA  section  furnishes  real-time 
feedback  on  the  capability  of  the  maintenance  per¬ 
sonnel  to  perform  their  tasks.  Our  data  shows  that 
the  QA  program  is  viable  and  doing  its  job  of  insuring 
that  maintenance  is  properly  performed. 


•  TASK  2:  Our  analysis  in  this  task  used  three  techniques 
to  examine  productivity.  Each  technique  produced  some 
positive  analytical  results.  The  approaches  used  were: 
examine  productivity  by  action  code,  examine  productiv¬ 
ity  by  frequency,  and  use  of  analysis  of  variance,  or 
ANOVA . 

The  productivity  by  action  code  approach  allowed 
us  to  examine  the  effect  training  had  on  produc¬ 
tivity  in  a  graphical  form.  The  results  appeared 
to  show  that  for  both  of  the  work  unit  codes  (WUCs) 
examined  (primary  flight  control  electronics  - 
14A00  and  turbofan  power  plant  -  23ZOO),  the  effect 
of  training  is  significant  in  terms  of  productivity 
increases.  It  was  obvious  in  our  "v/ing-to-wing " 
comparisons.  The  attempt  to  group  work  centers  by 
training  status  (high,  medium,  or  low)  and  thus 
infer  some  meaning  concerning  the  effect  of  fre¬ 
quency,  did  not  provide  useful  results.  Any  re¬ 
lationship,  holding  training  relatively  constant, 
between  frequency  and  productivity  was  not  obvious. 

-  The  productivity  by  frequency  (actions  per  worker) 
approach  plotted  frequency  versus  productivity.  A 
regression  line  was  fitted  to  each  plot  and  the 
results  for  WUC  14A00  showed,  in  four  out  of  six 
cases  (six  action  codes),  a  positive  correlation 
(negative  slope)  between  frequency  and  productivity 
The  results  for  WUC  23Z00  were  not  clear.  We  exam¬ 
ined  three  action  codes  and  in  two  cases  got  a 
negative  correlation  between  frequency  and  pro¬ 
ductivity.  One  case  resulted  in  a  positive  cor¬ 
relation.  Overall,  the  technique  appears  to 

show  a  positive  relationship  between  frequency 
and  productivity. 

-  The  use  of  ANOVA  allowed  us  to  examine  the  impact 
of  both  frequency  and  training  on  maintenance 
productivity.  The  statistical  results  were  mixed, 
since  in  three  out  of  four  tests  it  was  not  in¬ 
dicated  that  these  results  were  indicative  of  the 
overall  Air  Force  maintenance  population  at  the 
90%  confidence  level.  However,  it  must  be  noted 
that  training,  in  both  WUC  examinations,  has  a 
much  larger  effect  on  productivity  than  frequency. 
In  the  case  of  WUC  23Z00,  there  was  a  positive 
indication  of  a  relationship  between  training 

and  productivity  at  the  90%  confidence  level. 

In  order  to  assure  ourselves  that  the  amount  of 
time  spent  by  work  centers  on  the  actions  we 
examined  was  representative,  we  did  a  limited 
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comparison  of  actions  we  examined  to  total 
actions  worked  on.  There  are  28  system  level 
WUCs  for  the  F-16.  WUC  14000  was  the  highest 
manhour  consumer  in  our  sample  (10.6%),  WUC 
23000  was  fourth  with  8.0%.  There  are  seven  sub¬ 
systems  within  WUC  14000 — WUC  14A00  was  32%  or 
3.4%  of  total  wing  manhours.  There  are  twelve 
subsystems  within  WUC  23000 — WUC  23Z00  was  24% 
of  WUC  23000  or  2.0%  of  total  wing  manhours. 

Thus  out  of  113  subsystem  WUCs  the  two  WUCs  we 
examined  ( 14A00  and  23Z00)  are  quite  representa¬ 
tive  of  total  wing  maintenance  since  they  con¬ 
sume  over  5%  of  total  maintenance  manhours  in 
the  sample  we  looked  at. 

-  Our  intention  was  to  show  a  relationship  between 
maintenance  productivity  and  installation-level 
training.  We  chose  courses  taught  using  simula¬ 
tors  for  our  examination  of  training  but  did  not 
compare  simulator  training  with  non-simulator 
training . 

•  TASK  3:  In  this  task,  data  limitations  reduced  the 

scope  of  any  conclusions  that  could  be  made  with  respect 
to  the  results  of  our  analyses.  No  specific,  quantita¬ 
tive  observations  or  conclusions  can  be  advanced  concern¬ 
ing  the  relationship  between  installation-level  main¬ 
tenance  training  and  productivity.  Subjectively,  in¬ 
stallation-level  training  does  seem  to  have  a  positive 
impact  on  maintenance  productivity.  Interviews  were 
conducted  with  several  individuals  (ranging  from  mech¬ 
anics  to  staff  officers  at  the  divisional  level).  All 
of  these  individuals  had  the  same  impression  of 
installation-level  training:  although  the  positive 
benefit  of  the  training  may  not  be  quantifiable,  the 
benefit  does  exist.  Mechanics  were  able  to  "diagnose 
problems  better"  and  "perform  troubleshooting  actions 
with  more  accuracy"  as  the  result  of  installation-level 
training  (in  this  case,  Detriot  Diesel  Allison  courses). 

These  subjective  observations  are  all  that  can  be 
said,  at  this  time,  concerning  the  relationship  upon 
which  this  task  has  focused.  Current  databases  from 
which  information  can  be  obtained  for  training/produc¬ 
tivity  analyses  proved  inadequate  for  a  specific,  quan¬ 
titative  analysis.  Although  the  current  databases  are 
not  appropriate  for  the  kinds  of  analyses  that  we 
attempted,  this  will  not  necessarily  be  the  case  in 
the  future.  The  Army  is  developing  the  systems  to 
keep  track  of  productivity  information.  When  they 
are  completed,  the  present  type  of  analysis  could  be 
successful.  In  particular,  the  following  data  sources 
could  provide  appropriate  information. 
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-  The  Maintenance  Performance  System  (MPS).  The 
Army  Research  Institute-developed.  MPS  is  currently 
in  the  test  mode.  As  more  data  is  collected 

by  this  system,  and  if  the  system  is  expanded  so 
that  data  is  collected  at  other  Army  installations 
the  system  could  prove  to  be  a  very  effective 
training  management  tool,  especially  for  the  anal¬ 
yses  of  training  and  maintenance  productivity. 

-  The  Standard  Army  Maintenance  System  (SAMS) .  The 
SAMS  is  an  automated  maintenance  management  system 
that  will  replace  The  Army  Maintenance  Management 
System  (TAMMS)  and  encompass  all  levels  of  Army 
material  maintenance.  SAMS  will  improve  upon  the 
present  TAMMS  system  in  that  a  maintenance  job 
will  be  "tracked"  on  an  in-shop  computer  as  it 
progresses  through  work  stages,  and  each  different 
stage  of  work  will  be  explicitly  noted  in  the  job 
record.  Therefore,  the  records  should  be  more 
accurate  than  those  in  TAMMS  (which  is  automated 
at  a  much  higher  level)  and  include  more  detailed 
data  on  particular  actions  performed.  Unfortunate 
ly,  the  SAMS  system  is  not  designed  for  training 
analysis  purposes;  no  information  that  identifies 
individuals  is  included  in  this  system.  The  sys¬ 
tem,  however,  is  still  in  preliminary  implement¬ 
ation  stages.  Data  elements  could  theoretically 
be  added  to  the  system  if  a  strong  rationale  were 
given  for  their  inclusion.  Even  if  individual 
identification  were  not  included  in  the  system, 
the  improvements  in  accuracy  and  level  of  detail 
over  the  TAMMS  database  could  be  of  benefit  for 
training/productivity  analyses.  A  "macro"  level 
approach,  which  specifically  identifies  certain 
types  of  installation-level  training  with  certain 
types  of  maintenance  actions,  would  be  much  eas¬ 
ier  to  accomplish  if  maintenance  actions  were 
identified  more  explicitly  in  an  automated  data¬ 
base.  SAMS  could  provide  this  capability,  where¬ 
as  the  current  TAMMS  does  not. 

TASK  4;  The  Navy  training  pipeline  is  complex.  Prior 
to  fleet  assignment,  a  new  enlistee  might  attend  as 
many  as  seven  courses  located  at  different  schools. 
Mixing  self-paced  and  group-paced  fixed-length  courses 
in  the  same  pipeline  can  cause  scheduling  problems  and 
student  backlogs.  A  student  may  accelerate  through 
one  series  of  self-paced  courses,  only  to  have  to  wait 
for  a  start  date  for  the  next  course  if  it  is  group- 
paced.  The  efforts  of  one  school  to  solve  its  student 
backlog  problem  could  contribute,  however,  to  a  stu¬ 
dent  backlog  for  the  follow-on  course  at  another  school 


In  addition  to  alleviating  student  backlogs,  coordin¬ 
ation  can  eliminate  redundant  or  inadequate  instruction 
and  can  help  reduce  attrition  and  time  spent  in  train¬ 
ing.  The  electronics  technician  (ET)  pipelines  reflect 
several  instances  of  apparently  redundant  training.  To 
reinforce  the  fundamental  skills  taught  in  the  basic 
electricity  and  electronics  course,  the  Class  "A" 
schools  teach  basic  electrical  principles  and  refresh¬ 
er  mathematics.  Although  this  training  appears  re¬ 
dundant  and  increases  the  time  spent  in  school,  it  has 
reduced  attrition  at  the  follow-on  courses  by  rein¬ 
forcing  necessary  fundamental  skills.  Overall  pipe¬ 
line  attrition  is  another  factor  of  considerable  im¬ 
portance  in  the  examination  of  training  for  critical 
skills.  The  estimated  FY82  cohort  attrition  percen¬ 
tages  for  the  six  Navy  training  pipelines  included  in 
this  analysis  were  computed  and  are  summarized  below: 

Aviation  Electronics  Technician  --  19.7% 

Electronics  Technician/ Advanced 
Electronics  Field/Strategic  Weapons 
Systems  Submariner  —  40.4% 

Electronics  Technician/Nuclear  Field  —  57.6% 
Electronics  Technician/ Advanced 

Electronics  Field/Conventional  Surface  --  39.5% 

Electronics  Technician/Advanced  Electronics 
Field/Navigation  Submariner  --  50.5% 

-  Electronics  Technician/ Advanced  Electronics 

Field/Electronics  Warfare  Submariner  --  40.1% 

Overall  attrition  figures  could  key  Navy  planners  to 
problems  in  the  pipeline  as  a  whole,  as  opposed  to 
specific  courses  within  a  particular  pipeline.  If 
overall  attrition  figures  are  deemed  to  be  too  high, 
then  efforts  should  be  made  to  determine  the  exact 
cause  of  the  attrition.  Perhaps  entrance  requirements 
for  the  rating  under  consideration  need  to  be  raised 
or  courses  need  to  be  re-evaluated.  This  would  ensure 
that  training  funds  are  expended  in  a  fashion  that 
yields  the  highest  number  of  qualified  sailors  at  the 
end  of  the  training  pipeline. 

The  Navy  has  initiated  efforts  to  improve  pipeline 
management  and  reduce  the  time  spent  at  Navy  schools. 
Special  attention  has  been  given  to  those  skill  areas 
requiring  electronics  training.  One  key  to  better 
pipeline  management  is  a  simpler  pattern  of  training: 
keep  the  number  of  courses  and  the  various  school  lo¬ 
cations  to  a  minimum. 
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I.  INTRODUCTION 

This  report  describes  the  results  of  a  research  project 
to  perform  skill  training  analyses  in  the  various  Services;  in 
particular,  to  develop  a  linkage  between  individual  training  con¬ 
ducted  at  the  installation  level  and  unit  productivity.  This 
work  was  sponsored  by  the  Office  of  the  Assistant  Secretary  of 
Defense  (Manpower,  Reserve  Affairs  and  Logistics).  This  section 
describes  the: 

•  Purpose/Scope  of  Research, 

•  Background, 

•  General  Approach,  and 

•  Organization  of  this  Report. 

A.  PURPOSE/SCOPE  OF  RESEARCH 

The  overall  purpose  of  this  project  was  to  examine  Service 
skill  training  conducted  at  the  installation  level  and  to  analyze 
its  impact  on  unit  productivity.  The  MCR  research  required  gath¬ 
ering  information  on  both  skill  training  and  means  of  measuring 
maintenance  productivity.  MCR  also  conducted  a  special  purpose 
analysis  on  the  Navy  Pipeline  Management  System  with  emphasis  on 
a  specific  critical  skill.  This  research  is  documented  in  Vol¬ 
ume  II  of  this  report. 

Our  analysis  involved  four  different  tasks: 

•  an  examination  of  the  impact  of  a  Field  Training  De¬ 
tachment  ( FTD )  on  Air  Force  operational  unit  produc¬ 
tivity, 

•  an  examination  of  the  impact  of  installation-level 
training  using  simulators  on  F-16  unit  maintenance 
productivity. 


•  an  examination  of  the  impact  of  installation-level 
training  on  Army  operational  unit  maintenance  produc¬ 
tivity,  and 

•  an  examination  of  the  Navy  Pipeline  Management  System. 

B.  BACKGROUND 

OASD  ( MRA&L )  has  issued  policy  guidance  that  manpower 
impacts  will  be  considered  in  system  design  and  operation. 

This  consideration  requires  both  OSD  and  the  Services  to  develop 
improved  methods  for  evaluating  manpower,  personnel,  and  train¬ 
ing.  The  analysis  of  critical  skills,  particularly  the  impact 
of  maintenance  manpower  skills,  is  a  key  factor  in  these  evalu¬ 
ations.  Individuals  who  are  required  to  operate  and  maintain 
weapon  systems  reach  the  journeyman  level  through  individual 
training  conducted  at  the  installation  level. 

Due  to  a  growing  concern  for  the  problems  in  maintaining 
and  operating  existing  weapons  systems,  a  comprehensive  review 
of  training  was  initiated  by  OASD  (MRA&L) .  Some  initial 
work  has  been  completed  concerning  how  weapon  systems  support 
can  be  improved  with  better  training.  In  their  first  report—^, 
MRA&L  describes  existing  on-the-job  training  (OJT)  programs  and 

proposes  ways  to  enhance  training,  especially  OJT,  in  order  to 

2/ 

improve  equipment  maintenance.  In  a  later  report—  ,  MRA&L 

1/Report  on  the  OJT  Study~Task:  On-The-Job  Training  in  the 

Department  of  Defense,  OASD  (MRA&L)  under  the  auspices  of  the 
Defense  Education  and  Training  Executive  Committee,  January 
1981. 

2/Report  on  Individual  Skill  Training:  Maintenance  Training  in 
the  Department  of  Defense,  OASD  (MRA&L),  May  1982. 


provides  an  overview  of  the  systems  developed  to  train  indivi¬ 
duals  in  selected  maintenance  skills.  The  report  covers  the  en¬ 
tire  training  plan,  from  initial  skill  training  in  formal  schools 
to  individual  skill  training  received  in  the  unit.  Based  on  these 
completed  and  on-going  study  efforts,  and  in  support  of  the  MRA&L 
Training  and  Education  Directorate,  MCR  has  undertaken  the  present 
analysis  to  develop  a  linkage  between  individual  training  conducted 
at  the  installation  level  and  unit  productivity. 

C.  GENERAL  APPROACh 

In  our  analysis  of  the  linkage  of  unit  level  skill  training 
and  productivity,  our  focus  was  on  maintenance.  The  approach 
we  followed  was  to  choose  specific  highly  visible  skills  and  to 
examine  the  training  provided  and  the  related  maintenance  tasks 
performed.  This  required  an  initial  research  effort  to  find 
appropriate  skills  that  required  installation-level  training  or 
for  which  training  was  provided.  Also,  the  initial  research 
effort  focused  on  available  sources  of  maintenance  measures  of 
productivity.  The  intent  of  our  analytical  effort  was  to  esta¬ 
blish  specific  quantitative  linkages  between  skill  training  pro¬ 
vided  by  units  and  maintenance  productivity. 

D.  ORGANIZATION  OF  THIS  REPORT 

The  next  three  sections,  II-IV,  each  describe  one  of  three 
research  tasks  we  performed  (a  fourth  task  is  described  in  Vol¬ 
ume  II  of  this  report): 


•  An  Examination  of  the  Impact  of  a  Field  Training 
Detachment  (FTD)  on  Air  Force  Operational  Unit  Produc¬ 
tivity  ; 

•  An  Examination  of  Installation-Level  Training  Using 
Simulators  on  F-16  Unit  Maintenance  Productivity;  and 

•  An  Examination  of  the  Impact  of  Installation-Level 
Training  on  Army  Operational  Unit  Maintenance  Produc¬ 
tivity  . 

Each  section  has  a  subsection  describing  the  approach  or 
evaluation  we  used  and  a  subsection  describing  our  results. 

There  are  also  four  appendices: 

•  Reference  Sources, 

•  Summary  Outputs  from  Methodological  Data  Bases, 

•  Back-up  Data,  and 

•  Training  Effectiveness  Analysis  Background  Materials. 


II.  AN  EXAMINATION  OF  THE  IMPACT  OF  A 
FIELD  TRAINING  DETACHMENT  ( FTD)  ON 
AIR  FORCE  OPERATIONAL  UNIT  PRODUCTIVITY 


This  section  discusses  the  impact  of  an  FTD  on  maintenance 
performance  at  operational  Air  Force  units.  The  analysis  will 
develop  a  linkage  between  individual  training  conducted  at  the 
installation  level  and  unit  productivity.  Training  effectiveness 
must  be  measured  in  terms  of  on-the-job  performance  of  personnel. 
The  approach  to  this  research  task  included: 

•  identifying  appropriate  measures  of  performance  at 
the  wing  level;  and 

•  developing  methods  for  analyzing  these  measures  in  the 
context  of  the  problem  (i.e.,  the  impact  of  FTD 
training) . 

The  study  focused  on  maintenance  skills  which  are  critical 
to  the  operational  readiness  of  complex  weapon  systems.  The 
study  proceeded  in  three  stages. 

•  A  brief  review  of  published  research  on  initial  skill 
training  was  accomplished. 

•  Field  visits  were  made  to  operational  units  and  their 
supporting  FTDs  to  examine  unit  training  at  the  instal¬ 
lation  level.  This  field  work  included  interviews, 
observations  of  normal  operations  on  the  job,  and 

the  gathering  of  sample  data  for  development  of  our 
methodology . 

•  The  methodology  was  refined,  additional  data  was  col¬ 
lected,  and  the  results  were  analyzed. 

This  section  contains  two  subsections: 

•  Skill  Training  Evaluation,  and 

•  Skill  Training  Evaluation  Results. 


Section  II. A,  Skill  Training  Evaluation,  discusses  individual 
training  with  emphasis  on  installation  schools;  special  consider¬ 
ation  in  relating  maintenance  performance  to  training;  sample 
selection;  and  two  skill  training  evaluation  methodologies. 

Section  II. B,  Skill  Training  Evaluation  Results,  discusses 
application  of  the  two  skill  training  evaluation  methodologies; 
analysis  of  the  results  obtained;  and  observations. 

Supporting  data  is  presented  in  Appendix  B,  Summary  Outputs 
from  Methodological  Databases. 

A.  SKILL  TRAINING  EVALUATION 

The  focus  of  this  study  is  the  impact  of  training  conducted 
at  the  installation  level  on  maintenance  job  performance.  The 
typical  training  flow  for  Air  Force  maintenance  personnel  is 
shown  on  Exhibit  II-l.  These  enlistees  attend  initial  entry 
training  (basic  military  training  and  technical  school)  prior 
to  assignment.  During  technical  school,  they  will  learn  the 
basic  skills  required  within  their  Air  Force  Specialty  Code 
( AFSC ) .  Next,  they  receive  equipment-specific  training  in  an 
installation  Field  Training  Detachment  ( FTD) .  This  is  followed 
by  on-the-job  training  (OJT)  in  the  unit. 

This  section  discusses  the  following  topics; 

•  individual  training  with  emphasis  on  installation 
schools  and  their  relationship  to  the  Air  Force  wing 
maintenance  structure; 

•  special  considerations  involved  in  relating  maintenance 
performance  to  FTD  training; 

•  sample  selection;  and 
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•  MCR's  two  skill  training  evaluation  methodologies, 
which  are  the: 

Quality  Assurance  Methodology,  and  the 
-  Work  Unit  Code/Trend  Analysis  Methodology. 

1 .  Installation  Schools 

The  primary  functions  of  an  installation  school  include 
the  following: 

•  to  supplement  the  generalized  training  received  in 
formal  schools  with  equipment-specific  training 
related  to  types  of  equipment  located  at  that 
installation; 

•  to  provide  transition  training  to  personnel  whose 
previous  experience  has  been  on  other  models  of 
equipment;  and 

•  to  provide  upgraded  and  refresher  training. 

There  are  two  types  of  installation  schools:  officially 
recognized  schools  and  unofficial  schools.  Officially  recog¬ 
nized  schools  are  specifically  funded  and  are  manned  in  accord¬ 
ance  with  authorization  documents.  Although  these  schools  may 
be  affiliated  with  Service  training  commands,  their  main  purpose 
is  to  meet  the  specific  needs  of  the  associated  unit  at  the 
installation.  Included  among  the  officially  recognized  schools 
are  the  Air  Force  Field  Training  Detachments,  which  focus  on 
aviation  maintenance  training. 

Unofficial  installation  schools  are  established  by  an 
operational  unit,  using  its  own  manpower  and  funding  resources. 

In  the  Air  Force,  unofficial  schools  are  used  extensively  in 
the  Tactical  Air  Command  (TAC).  The  unofficial  schools  in  TAC 
provide  training  in  addition  to  that  provided  by  the  FTDs . 


Most  of  the  installation  school  training  in  the  Air 
Force  takes  place  at  FTDs .  Exhibit  II-2  shows  the  organizational 
relationship  between  the  Air  Training  Command  and  the  FTDs. 

There  are  49  FTDs  assigned  to  the  Strategic/Airlif t  Branch  of 
the  Field  Training  Group,  and  40  are  assigned  to  the  Tactical 
Branch.  Each  FTD  supports  at  least  one  unit. 

The  research  documented  by  this  section  of  the  Technical 
Report  focused  on  FTDs  assigned  to  the  Tactical  Branch  which  sup¬ 
ports  TAC.  Exhibit  II-3  shows  the  relationship  between  the  main¬ 
tenance  structure  of  a  tactical  wing  and  its  supporting  FTD.  Typi¬ 
cally,  there  are  three  or  four  flying  squadrons  in  a  wing.  The 
Deputy  Commander  for  Maintenance  ( DCM)  manages  equipment  main¬ 
tenance  for  these  flying  squadrons.  The  DCM  is  also  responsible 
for  assuring  that  maintenance  training  (FTD,  OJT,  and  informal 
programs)  is  effective.  Maintenance  work  is  performed  by  work 
centers  in  the  Aircraft  Generation  Squadron  (AGS),  Component  Re¬ 
pair  Squadron  (CRS),  and  Equipment  Maintenance  Squadron  (EMS). 

Some  work  centers  are  manned  primarily  by  personnel  trained  in  a 
single  skill  area  (e.g.,  all  jet  engine  mechanics),  while  others 
are  manned  by  personnel  trained  in  several  skill  areas  (e.g., 
integrated  avionics  specialists,  aircraft  electrical  systems 
specialists,  and  jet  engine  mechanics).  Team  maintenance  (two 
or  more  individuals  performing  a  single  maintenance  task)  is  stan¬ 
dard  procedure  in  most  wings.  Cross-skill  maintenance  (personnel 
who  are  trained  in  one  skill  area  performing  a  maintenance  task 
associated  with  a  different  skill  area)  is  performed  as  required 
by  workload  and  personnel  shortages. 
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HEADQUARTERS  (IIQ) 
AIR  TRAINING  COMMAND 


Exhibit  I 1-2.  AIR  TRAINING  COMMAND  - 
FIELD  TRAINING  DETACHMENT  RELATIONSHIP 


Special  Considerations 

Comparative  analysis  is  the  key  to  assessing  the  impact 
of  training  on  job  performance.  This  study  attempts  to  measure 
the  impact  of  FTD  training  by  comparing  the  performance  of  more- 
trained  maintenance  personnel  with  less-trained  maintenance  per¬ 
sonnel  in  the  same  skill  area.  In  addition  to  training,  several 
other  factors  (e.g.,  experience  levels  and  quality  of  super¬ 
vision)  affect  maintenance  performance,  but  are  not  specifically 
addressed  in  this  study. 

There  are  several  considerations  involved  in  relating 
maintenance  performance  to  amount  of  training: 

•  description  of  equipment, 

•  maintenance  performed, 

•  training  received,  and 

•  identification  of  maintenance  personnel. 

Comparisons  of  performance  can  be  made  only  when  similar  models 
of  equipment  are  involved.  The  equipment  and  the  maintenance 
task  performed  must  be  identified  at  a  level  which  is  specific 
enough  to  relate  the  maintenance  performed  to  a  single  skill 
area  and  skill  level.  This,  in  turn,  can  be  related  to  specific 
types  and  amounts  of  training. 

In  addition,  the  individuals  must  be  designated  in  a 
manner  that  identifies  their  skill  areas  and  the  training  they 
have  received.  When  individuals  performing  similar  maintenance 
tasks  can  be  identified,  comparisons  can  be  made  of  the  per¬ 
formance  of  personnel  with  different  degrees  of  training  in 


Assessment  of  training  effectiveness  is  more  difficult. 


however,  when  only  the  organizations  (e.g.,  work  centers)  per¬ 
forming  the  maintenance  (and  not  the  specific  individuals)  are 

identified.  The  maintenance  organizations  of  all  tactical 

3  / 

wings  report  through  the  same  management  system—  and  have 
similar  work  center  structures.  If  corresponding  work  centers 
in  different  wings  can  be  characterized  in  terms  of  the  training 
and  experience  levels  of  their  personnel,  then  comparisons  can 
be  made  of  the  work  centers*  performances  with  respect  to  the 
same  maintenance  task  on  the  same  type  of  equipment.  However, 
the  precise  relationship  between  work  center  differences  in 
performance  and  in  FTD  training  is  not  clear-cut.  There  are 
several  problems  with  determining  this  relationship.  First, 
the  method  of  maintenance  (i.e.,  team,  cross-skill,  or  indi¬ 
vidual)  is  often  difficult  to  identify.  Second,  the  actual 
amount  of  work  performed  by  each  individual  member  of  a  work 
center  is  not  easy  to  determine.  Third,  the  impact  of  FTD 
training  should  be  most  evident  on  newly  trained  personnel  (due 
to  less  OJT  and  less  experience) ,  yet  these  personnel  have 
relatively  little  effect  on  overall  work  center  team  perfor¬ 
mance.  Last,  the  maintenance  experience  and  formal  training 
received  prior  to  FTD  training  differs  among  individuals. 

3 .  Sample  Selection 

To  insure  homogeneity  of  the  data,  the  following 
criterion  was  used  to  select  our  sample:  select  a  representative 

2/De  fined  in  Air  Force  Regulation  6b-l,  Equipment  Maintenance: 

Maintenance  Management . 
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sample  of  data  that  minimizes  the  number  of  aircraft  types  ana 
geographic  locations  involved  in  the  data  collection  effort.  Se¬ 
lection  of  the  F-4  and  F-15  aircraft  allowed  for  a  sufficient  mix¬ 
ture  of  data  to  be  collected  from  only  three  geographic  locations, 
as  follows: 

•  FTD  201,  Langley  AFB,  VA 

-  F-15  List  Tactical  Fighter  Wing  (TFW)J 

•  FTD  516,  George  AFB,  CA 

F-4  (35th  TFW ,  37th  TFW) 

•  FTD  527,  Luke  AFB,  AZ 

F-4  L58th  Tactical  Training  Wing  (TTW)] 

F-15  (405th  TTW ) 

During  the  data  collection  phase  of  this  study,  it  was  learned 
that  the  58th  TTW  was  converting  from  F-4  aircraft  to  F-16  air¬ 
craft.  Since  necessary  F-4  data  was  no  longer  available  at 
that  wing,  the  58th  TTW  was  deleted  from  the  study.  In  addition, 
the  35th  TFW  was  eliminated  from  the  study  because  of  extreme 
delays  in  receiving  data. 

The  identity  of  the  wings  we  analyzed  is  not  relevant; 
hence,  we  have  randomly  titled  them  as  Wing  A,  Wing  B,  and  Wing 
C.  They  will  be  consistently  referred  to  by  these  names  in 
Section  II. B,  Skill  Training  Lvaluation  Results. 

4.  Methodology 

As  the  first  step  in  developing  skill  training  evaluation 
methodologies,  quantifiable  measures  of  maintenance  performance 
were  formulated  from  existing  historical  data  bases.  One  mea- 


sure  is  maintenance  man-hours  per  maintenance  task,  where  "task" 
is  a  single  procedure  performed  on  a  piece  of  equipment  (e.g., 
removal  of  a  component) .  The  information  is  maintained  by  the 
Air  Force  Maintenance  Data  Collection  (MDC)  system  based  on 
information  routinely  submitted  by  work  centers.  A  second  mea¬ 
sure  identified  is  the  rating  received  by  an  individual  during 
a  quality  assurance  (QA)  personnel  evaluation.  In  such  an  evalu¬ 
ation,  an  individual  is  rated  as  satisfactory  or  unsatisfactory 
by  quality  assurance  personnel  while  performing  a  maintenance 
task.  The  rating  applies  to  the  specific  task  evaluated,  and 
not  to  other  tasks  the  individual  is  qualified  to  perform. 

Each  wing  has  a  Quality  Assurance  Section  which  develops  a 
locally-computed  standard  baseline  for  specific  actions.  This 
standard  is  based  on  historical  data  and  the  ratings  assigned  to 
evaluations  are  derived  by  comparing  the  number  of  discrepancies 
(errors)  in  various  categories  to  the  baseline.  Both  the  MDC 
system  and  QA  evaluations  are  addressed  in  detail  in  Air  Force 
Regulation  66-1  (Volume  2),  Equipment  Maintenance;  Maintenance 
Management . 

The  remainder  of  this  section  discusses: 

•  Quality  Assurance  (QA)  Skill  Training  Evaluation 
Methodology,  and 

•  Work  Unit  Code/Trend  Analysis  (WUC/TA)  Skill  Training 
Evaluation  Methodology. 


a .  Quality  Assurance  (QA)  Skill  Training  Evaluation 
Methodology 

The  methodology  described  in  this  section  is  a  five 
step  decision-making  process  which  is  shown  in  Exhibit  II-4  and 
is  explained  in  more  detail  below. 

Step  1.  Request  QA  personnel  evaluation  data  from 
each  wing  examined. 


Quality  Assurance  personnel  evaluations  provided  the 
information  needed  for  this  method  of  skill  training  evaluation. 

■ 

Personnel  are  evaluated  while  performing  a  specific  action  by 
QA  teams  and  are  assigned  a  rating  of  pass  or  fail,  depending 
on  the  number  of  errors  detected.  The  three  wings  that  were 
examined  in  this  study  routinely  convert  QA  evaluation  data  into 
a  standard  format  and  store  the  data  in  computer  files.  This 
data  availability  and  format  provided  two  advantages. 

•  A  large  sample  would  easily  be  available  for  analysis. 

•  The  data  requested  could  be  restricted  to  a  specific 
time  period. 

■ 

An  individual's  training  status  is  constantly  changing,  so  a 
data  request  from  a  specific  time  period  allowed  for  a  pre¬ 
cise  link  with  the  needed  training  information  for  that  time 
period. 

Step  2.  Request  Course  Status  Summaries  from  each 
wing . 

In  order  to  link  QA  personnel  evaluations  with 
training  information,  we  required  outputs  from  the  training  sub¬ 
system  of  the  Maintenance  Management  Information  and  Control 


Exhibit  I I- 4 .  QA  DECISION  MAKING  PROCESS 
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System  (MMICS).  Course  Status  Summaries  for  particular  Field 
Training  Detachment  (FTD)  courses  were  requested  from  each  wing, 
and  the  information  in  these  summaries  was  used  to  determine  the 
individual  trainings  satus  for  personnel  listed  on  the  QA  report. 

Not  all  Course  Status  Summaries  from  each  wing  were  obtained, 
however,  and  some  individual  QA  evaluations  were  eliminated  as 
a  result.  Sample  QA  data  from  the  1st  TFW  provided  the  basis 
for  the  criteria  by  which  a  Course  Status  Summary  was  requested. 

If  a  specific  AFSC  was  not  represented  or  was  sparsely  represented 
in  the  QA  data,  then  all  FTD  courses  pertaining  to  that  AFSC 
were  eliminated  from  the  course  report  list.  This  data  use  limit¬ 
ation,  however,  had  a  negligible  effect  on  overall  QA  data  sample 
size.  Our  final  sample  used  90.4%  of  all  QA  observations  that 
were  originally  obtained. 

Step  3.  Determine  individual  training  status  for 

each  QA  observation.  \ 


Once  QA  data  and  Course  Status  Summaries  were  obtained 
for  the  three  wings  in  our  sample,  training  status  determinations 
had  to  be  made  for  each  QA  observation.  For  the  purpose  of  this 
analysis,  individuals  were  classified  as  trained,  untrained,  or 
OJT  trained.  These  determinations  are  defined  as  follows. 

•  Trained  (T)--in  order  for  an  individual  to  be  considered 
"trained"  in  this  analysis,  that  individual  must  have 
completed  all  courses  that  were  required  of  him. 

•  Untrained  (U) — an  individual  is  labeled  "untrained" 

in  this  analysis  if  he  is  listed  as  "awaiting  training" 
for  any  course  that  is  required  for  him. 
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•  OJT  Trained  (OJT)--if  an  individual  is  not  listed  for 
any  courses  on  the  Course  Status  Summaries,  either  as 
awaiting  completion  of  training  (AWACT)  or  completed 
training  (COMPL),  then  he  is  considered  OJT. 

"Required"  courses  have  different  meanings  in  the 
different  wings  examined.  In  the  1st  and  37th  TFWs,  courses 
are  considered  to  be  required  for  an  individual  if  he  is  listed 
for  the  course  in  any  way,  either  AWACT  or  COMPL.  However,  in 
the  405th  TTW,  course  requirements  are  listed  for  each  work 
center.  Personnel  assigned  to  a  specific  work  center  are  re¬ 
quired  to  take  all  courses  that  are  pertinent  to  the  operation 
of  that  work  center.  Exhibits  II-5  (1st  TFW/37th  TFW)  and  II-6 
(405th  TTW)  illustrate  the  process  for  determining  individual 
training  status  for  QA  observations  in  the  wings  examined.  The 
process  relates  requirements  criteria  to  the  T/u/OJT  definitions 
to  determine  to  which  category  a  specific  QA  observation  is 
assigned.  This  process  was  followed  for  all  QA  observations  in 
the  three  wings  examined.  It  provided  a  database  in  which  indi¬ 
vidual  training  status  is  directly  linked  with  individual  job 
performance  (as  measured  by  a  QA  rating) . 


Step  4.  Categorize  QA  observations. 

This  step  transformed  the  data  into  a  form  that  lends 
itself  to  statistical  applications.  Each  QA  observation  is 
represented  as  being  in  one  of  six  training/performance  categories: 

•  trained/passed  QA  evaluation, 

•  untrained/passed  QA  evaluation, 

•  OJT  trained/passed  QA  evaluation, 


1st  TFM/ 37th  TFW 

Requirements";  If  a  person  is  listed  for  a  course 
in  any  way,  it  is  required. 


If  do  not  have  Course  Status  Summary 
information  for  a  particular  AFSC 
course,  delete  all  individuals  with 


Exhibit  I 1-6 .  INDIVIDUAL  TRAINING  STATUS 
DETERMINATION  PROCESS  (405th  TTW ) 


trained/failed  QA  evaluation, 
untrained/failed  QA  evaluatin,  and 
OJT  trained/failed  QA  evaluation. 


l 


* 

« 

. ! 

Step  5.  Perform  tests  on  all  QA  categories  to  .  ■< 

determine  the  statistical  significance 
of  training.  -- 

-  i 

The  observations  in  the  training/performance  categories 
were  used  to  perform  statistical  tests  on  the  effectiveness  of 
training  within  a  specific  wing  at  particular  training  levels. 

I  j 

Once  this  step  was  completed,  the  results  were  analyzed  and  ; 

observations  made  concerning  these  results. 


b .  Work  Unit  Code/Trend  Analysis  (WUC/TA)  Skill 

Training  Evaluation  Methodology 

This  methodology  evaluates  the  effect  training 
differences  have  on  the  time  a  work  center  takes  to  perform 
a  task.  This  methodology  was  developed  by  MC?  using  two  Air 
Force  data  sources: 

•  maintenance  information,  taken  from  the  Maintenance 
Data  Collection  (MDC)  System;  and 

•  training  information,  taken  from  the  training  subsystem 
of  the  Maintenance  Management  Information  and  Control 
System  (MMICS). 

There  are  seven  decision-making  steps  in  this 
process  as  shown  on  Exhibit  II-7.  Each  step  is  described  in  de¬ 
tail  below. 


Step  1.  Choose  n^gh  manhour  consuming  WUCs  for 
each  wing,  determine  overlap  WUCs,  and 
request  WUC/TA  reports. 

The  maintenance  information  used  in  this  method  was 

ijj 

based  on  data  recorded  by  each  work  center  on  Air  Force  form 
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Choose  high  manhour  consuming  WUCs  for  each 
wing,  determine  overlap  WUCs,  and  request 
VfUC/TA  reports. 


_ * _ 

Request  training  information  from  the  wings  being 
examined  based  on  a  determination  of  which 
AFSCs  work  on  each  WUC  action. 


X. _ 

Link  training  information  to  each  work  center 
group  performing  a  WUC  action.  Delete  WUC 
actions  for  which  training  data  is  not  available. 

_ I _ 

Compare  work  centers  involved  in  like  WUC 
actions.  Delete  actions  where  the  time  difference 
between  work  centers  is  greater  than  100%. 

~ . ~  "  '  I' . ' . " . 

Compare  work  centers  involved  in  like  WUC  actions. 
Calculate  time  ratio  for  each  comparison.  Calculat 
training  percentage  for  each  work  center  group. 


Aggregate  all  comparison  information. 


Perform  analytical  tests  and  determine  significance 
of  training. 


Exhibit  I 1-7.  WUC/TA  DECISION  MAKING  PROCESS 


AFTO  349,  "Maintenance  Data  Collection  Record,"  which  is  entered 
into  the  MDC  system.  The  standard  output  report  used  is  called 
the  Work  Unit  Code  (WUC)  Trend’  Analysis  (TA)  report.  The  WUC 
identifies  the  equipment  (e.g.,  F-100  engine)  on  which  maintenance 
is  performed  for  a  major  weapon  system  (e.g.,  F-15  aircraft).  The 
Action  Taken  (AT)  Code  identifies  what  work  was  done  on  the  equip¬ 
ment  (e.g.,  AT  code  R  for  "Remove  and  Replace").  The  WUC/TA  report 
lists  the  length  of  time  that  a  work  center  took  to  do  a  particu¬ 
lar  maintenance  action  on  a  particular  type  of  aircraft.  High- 
manhour  consuming  WUCs  were  chosen  so  as  to  ensure  a  large  sample 
of  typical  tasks.  "Overlap"  WUCs  were  used  so  that  inter-wing 
comparisons  could  be  made.  The  term  overlap  means  that  two  wings 
each  list  the  same  WUC  as  high-manhour  consuming. 

Step  2.  Request  training  information  from  the  wings 
being  examined  based  on  a  determination  of 
which  AFSCs  work  on  each  WUC  action. 

The  linkage  of  maintenance  action  performed  to 
individual  training  requires  a  knowledge  of  which  skills  ar’e  re¬ 
quired  to  work  on  which  pieces  of  equipment.  •  In  our  analysis, 
we  attempted  to  precisely  identify  the  skill  IAFSC)  that  is  asso¬ 
ciated  with  a  particular  maintenance  action.  For  example,  in 
the  Aircraft  Generation  Squadron  (AGS),  a  weapons  flight  consists 
of  AFSC  462X0  personnel  (weapons  mechanic) .  Therefore,  all  work 
performed  by  a  weapons  flight:  can  be  assumed  to  be  done  by  any 
of  their  trained  personnel.  On  the  other  hand,  the  specialist 
flights  of  the  AGS  include  personnel  with  a  variety  of  differ¬ 
ent  AFSOs.  Therefore,  only  specific  personnel  can  be  expected 
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to  do  particular  jobs.  For  example,  personnel  holding  AFSC  326X6 
(integrated  avionics  attack  control  specialist)  are  expected  to 
perform  the  action  to  replace  (R)  the  radar  transmitter  for  the 
fire  control  system  of  the  F-15  (WUC  74FAO) .  There  are  usually 
about  12  to  15  personnel  with  this  AFSC  in  the  F-15  AGS  specialist 
flight,  which  includes  an  approximate  total  of  90  personnel. 

In  order  to  request  the  proper  training  information 
for  each  AFSC,  it  is  necessary  to  determine  which  local  course(s) 
taught  by  the  FTD  apply  to  the  AFSC  and  WUC  under  examination. 
Continuing  with  the  example  above,  the  course  326X6-002  (F-15 
Fire  Control)  is  required  for  all  personnel  holding  that  AFSC 
to  be  considered  fully  trained  in  the  1st  TFW. 

Step  3.  Link  training  information  to  each  work 
center  group  performing  a  WUC  action. 

Delete  WUC  actions  for  which  training 
is  not  available. 

Next,  it  was  necessary  to  calculate  a  specific  number 
of  trained,  untrained,  and  on-the-job  trained  (OJT)  personnel  within 
each  work  center  for  each  specific  AFSC  performing  a  WUC  action 
under  consideration.  In  this  methodology,  the  terms  trained, 
untrained,  and  OJT  have  the  same  meaning  as  in  the  QA  procedure. 
Also,  the  linkage  between  training  information  and  individuals 
in  work  centers  who  are  performing  specific  actions  is  accom¬ 
plished  in  the  same  way  as  in  the  QA  methodology.  In  those 
cases  where  training  information  was  not  available,  the  WUC 
action  was  deleted. 
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Step  4.  Compare  work  centers  involved  in  like 
WUC  actions.  Delete  actions  where  the 
time  difference  between  work  centers  is 
greater  than  100  percent. 

Once  training  information  was  linked  with  WUC/TA 
data,  comparisons  between  work  center  groups  performing  like  ac¬ 
tions  were  made.  Where  two  or  more  work  centers  were  involved  in 
like  WUC  actions,  we  compared  the  times  associated  with  the 
action  performed.  When  the  time  difference  in  a  comparison  was 
greater  than  100  percent,  we  deleted  the  action.  The  observa¬ 
tions  deleted  were  included  and  tested  in  one  set  of  compari¬ 
sons  (1st  TFW)  and  did  not  change  the  results.  However,  it  was 
still  determined  that  including  observations  which  had  large 
differences  (over  100%)  would  not  be  analytically  sound.  These 
large  differences  appear  to  be  caused  by  something  in  the  quan¬ 
tity  or  quality  of  the  work  being  performed  that  is  not  readily 
apparent  and  not  the  result  of  training  differences.  In  exam¬ 
ining  work  center  groups,  training  data  (the  percentage  of 
personnel  classified  T,  U,  or  OJT  within  the  work  center  group) 
and  the  time  spent  in  doing  a  specific  action  were  used  in  each 
set  of  comparisons. 


Step  5.  Compare  work  centers  involved  in  like 
WUC  actions.  Calculate  time  ratio  for 
each  comparison.  Calculate  training 
percentage  for  each  work  center  group. 

In  each  comparison,  we  noted  which  work  center  was 
more  trained  and  which  work  center  was  less  trained.  We  then 
compared  the  WUC  action  times  of  these  work  centers  in  the 
following  ratio  form: 
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More-trained  work  center  action  time 
Less-trained  work  center  action  time 

This  ratio  expresses  the  more-trained  work  center's 
action  time  as  a  fraction  of  the  less-trained  work  center's 
action  time.  If  this  fraction  is  less  than  1,  the  more-trained 
work  center  performed  faster;  if  the  number  is  greater  than  1, 
the  more-trained  work  center  was  slower. 

The  reason  for  this  comparison  is  straightforward. 

A  difference  in  training  should  equate  to  a  difference  in  time 
spent  to  perform  a  task.  As  the  training  difference  increases, 
the  time  difference  should  also  increase.  In  this  case,  however, 
a  simple  "difference  in  time"  would  not  be  an  appropriate  com¬ 
parative  variable.  A  ratio  allows  for  a  more  straightforward 
comparison  with  training  differences  which  are  in  percentages. 

Comparisons  in  this  fashion  were  accomplished  for 
all  work  center  groups  performing  the  same  WUC  action  within  a 
wing  (intra-wing),  and  between  wings  that  have  the  same  type  of 
aircraft  (inter-wing).  Average  values  for  the  variables  were 
then  calculated. 

We  computed  an  average  value  for  the  following 
variables;  more-trained  work  center  training  percentage,  less- 
trained  work  center  training  pecentage,  and  work  center  action 
time  ratio.  These  values  were  computed  for  both  intra-wing  and 
inter-wing  work  center  comparisons. 

Step  6.  Aggregate  all  comparison  information. 

All  of  the  comparative  variables  were  added  together 
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and  averaged  to  derive  an  average  value  for  each  intra-wing  and 
inter-wing  set  of  comparisons. 

Step  7.  Perform  analytical  tests  and  determine 
significance  of  training. 

Our  results  should  allow  a  significance  test  to  be 
performed  at  this  final  step.  The  analysis  is  intended  to  show 
the  significance  of  training  on  maintenance  productivity  measured 
in  performance  time.  The  observations  and  resulting  comparisons 
were  analyzed  and  observations  made  concerning  them. 

B.  SKILL  TRAINING  EVALUATION  RESULTS 

This  section  discusses  the  following: 

•  application  of  the  QA  and  WUC/TA  skill  training 
evaluation  methodologies, 

•  analysis  of  QA  and  WUC/TA  results,  and 

•  observations  with  respect  to  results  that  were 
obtained. 

1 .  Application  of  Methodologies 

The  QA  and  WUC/TA  skill  training  evaluation  methodologies 
were  applied  to  data  from  three  wings:  1st  TFW,  37th  TFW,  and 
the  405th  TTW.  Our  original  plans  had  included  two  additional 
wings:  the  58th  TTW  and  the  35th  TFW.  The  58th  was  excluded 
because  the  wing  was  converting  from  F-4  to  F-16  aircraft.  The 
35th  TFW  was  excluded  because  data  was  not  available  until  late 
in  the  study,  and  our  sample  contained  enough  data  to  perform 
the  hypothesis  testing  necessary  to  determine  the  linkage  between 
installation  training  and  maintenance  performance. 
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a .  QA  Application 

The  QA  skill  training  evaluation  methodology  was 
applied  to  approximately  2,400  evaluations  from  the  three  wings 
that  were  examined.  With  the  elimination  of  those  evaluations 
for  which  there  was  no  FTD  course  data,  the  QA  database  con¬ 
tained  a  total  of  2,180  evaluations.  The  decision-making  pro¬ 
cess  previously  illustrated  was  utilized  to  associate  a  training 
status  with  each  of  their  evaluations.  Each  QA  personnel  evalu¬ 
ation  was  classified  into  a  training/performance  category  (e.g., 
trained/passed) . 

After  completing  the  classifications,  the  data 
for  this  analysis  was  in  a  form  that  lends  itself  to  statistical 
analysis.  An  appropriate  statistical  method  for  testing  data 
of  this  type  is  a  hypothesis  test  for  proportions.  An  example 
will  illustrate  the  test  procedure. 

In  this  type  of  a  problem,  two  populations  are 
under  consideration  (e.g.,  one  of  trained  individuals  and  one  of 
untrained  individuals).  Exhibit  II-8  displays  sample  data  for 
trained  and  untrained  individuals  from  the  QA  database. 


All  Comparisons  (All  Wings,  All  Skill  Levels) 


Number  of  Evaluations 


QA 

Evaluation 

Status 


#  Trained 


#  Untrained 


Pass 

Fail 

Total 


if  Total 


286 

786 

135 

395 

421 

1181 

Exhibit  1 1-8.  SAMPLE  DATA 
FROM  THE  QA  DATABASE 
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To  distinguish  the  true  populations  from  the  samples  taken  from 

the  two  populations,  the  following  variables  are  defined: 

=  proportion  of  the  sample  of  trained  individuals  who 
passed  their  QA  evaluation 
=  500  of  sample  who  passed  _ 


760  total  number  in  sample 


P 2  -  proportion  of  the  sample  of  untrained  individuals 
who  passed  their  QA  evaluation 
=  286  of  sample  who  passed  _ 

421  total  number  in  sample 

P^  =  proportion  of  the  true  population  of  trained  indi¬ 
viduals  who  passed  their  QA  evaluation 

P 2  =  proportion  of  the  true  population  of  untrained 
individuals  who  passed  their  QA  evaluation 

The  question  is  whether  or  not  the  hypothesis 
accePted  or  rejected  statistically.  If  the 
difference  between  the  two  sample  proportions  P^  and  P^ 
is  statistically  significant,  then  the  hypothesis  Hq  is 
rejected,  and  trained  individuals  can  therefore  be  considered 
more  likely  to  pass  QA  evaluations  than  untrained  individuals. 

In  order  to  perform  the  desired  hypothesis  test, 
values  for  P^  and  p ^  a re  needed.  Because  the  hypothesis  test¬ 
ed  is  ^q:Pi~P2'  only  one  va-*-ue  has  to  he  derived.  From  our 
data,  the  best  sample  estimate  for  P ^  and  p^  is  P,  or  the 

proportion  of  individuals  who  passed  their  QA  evaluation  from 

7  3G 

the  combined  sample  group.  In  this  example,  P  =  , 


or  .666. 


All  of  the  proportions  needed  to  test  the  hypothesis 


H  :  p  =  p„  are  now  available.  The  next  step  is  to  compute 
o  1  2 

the  standard  error  of  the  difference  which  i-s: 
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(1/N,  +  1/N ..  , 

1  <L 


y 


'  (  1  _  w'  ) 


Where  =  total  number 

of  evaluations 
of  trained  per¬ 
sonnel  (from 
Exhibit  II-8) . 

=  total  number 
of  evaluations 
of  untrained  per¬ 
sonnel  (from 
Exhibit  II-8) . 


Computing  this  value  gives  us: 


P  -P 
1  2 


666)  ( 1 - . 666)  ( 


760 


421 


.029 


This  standard  error  value  is  now  used  in  computing 
the  Z-statistic  for  the  hypothesis.  This  statistic  is  compared 
against  a  normal  distribution  at  a  specific  level  of  confidence 
to  determine  whether  or  not  the  hypothesis  P^~P2  should  be 
rejected. 


The  formula  for  the  Z-statistic  is: 


U 


2  = 


Pl-P2~  2NXN2 


where  N  =  total  number  of 

evaluations  of  all 
personnel; 

(from  Exhibit  II-8) . 


Computing  this  value  gives  us: 


(1181) 

.658  -  .679  -  (2)  (760)  (421) 

.029 


-.793 


At  a  90  percent  level  of  significance,  the  hypothesis 
Ho;  p\~  P  2  ;''s  rejecteci  only  if  the  Z-statistic  is  greater 
than  1.28.  In  this  case,  -.793  is  clearly  not  greater  than  1.28, 
so  the  hypothesis  cannot  be  rejected.  In  other  words, 
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this  statistical  result  does  not  show  that  trained  individuals 
are  more  likely  to  pass  a  QA  evaluation  than  untrained  individuals. 

Exhibits  I 1-9  and  11-10  show  the  QA  evaluation  compar¬ 
ison  data  and  the  statistical  results  for  each  set  of  comparisons 
made  using  the  QA  database.  Two  tests  were  performed  on  each 
category: 

•  one  comparing  trained  individuals  with  untrained 
individuals,  and 

•  one  comparing  both  trained  and  OJT  individuals 
with  untrained  individuals. 

These  two  types  of  comparisons  were  made  for  the  following 
reason.  The  first  attempts  to  determine  the  statistical  sig¬ 
nificance  of  FTD  course  training,  and  the  second  measures  the 
significance  of  no  training  at  all  ("untrained"  in  the  sense 
that  the  individual  has  not  been  formally  trained  at  the  FTD 
level  or  trained  on-the-job) .  In  most  cases,  the  test  results 
indicated  that  the  hypothesis  Hq:  p^=  cannot  be  rejected. 

b.  WUC/TA  Application 

The  WUC/TA  skill  training  evaluation  methodology 
was  applied  to  data  received  from  the  three  wings  in  our  final  sam¬ 
ple.  Exhibit  11-11  lists  the  WUCs  that  were  chosen  for  analysis. 

All  of  these  WUCs  are  in  the  top-25  manhour  consuming  components 
list  for  two  different  wings  that  use  the  same  aircraft.  A 
training  status  was  then  associated  with  this  WUC/TA  informa¬ 
tion  using  the  decision-making  process  previously  illustrated. 
Training  percentages  were  then  computed  for  each  work  center 
involved  in  overlap  actions. 
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Exhibit  I 1-9 .  QA  EVALUATION 
COMPARISON  DATA 


J 

fl  n 

a  _ 

a  " 

* 

a  I 

■  . 

* 

« 

COMPARISON 

GROUPING 

COMPARISON  SET 

STD. 

ERROR 

^P  -P 

1  2 

2 

STATISTIC 

[90%  LEVEL  01 
SIGNIFICANCI 
(Z  >1.28) 
3n  REJECTED 

OVERALL  COMPARISONS 
(ALL  WINGS,  ALL 

SKILL  LEVELS) 

TRAINED  TO  UNTRAINED 

.0287 

-.7931 

NO 

TRAINED  +  OJT  TO  UNTRAINED 

.0254 

-.1242 

NO 

WING  B-ALL 

SKILL  LEVELS 

TRAINED  TO  UNTRAINED 

.0455 

-.4164 

NO 

TRAINED  +  OJT  TO  UNTRAINED 

.0339 

1.4582 

YES 

WING  C-ALL 

SKILL  LEVELS 

TRAINED  TO  UNTRAINED 

.0561 

2.7780 

YES 

TRAINED  +  OJT  TO  UNTRAINED 

.0537 

2.6937 

YES 

WING  A-ALL 

SKILL  LEVELS 

TRAINED  TO  UNTRAINED 

.0718 

-3.0155 

NO 

TRAINED  +  OJT  TO  UNTRAINED 

.0649 

-2.5249 

NO 

WING  B 

SKILL  LEVEL  3 

TRAINED  TO  UNTRAINED 

.0833 

-1.7281 

NO 

TRAINED  4-  OJT  TO  UNTRAINED 

.0606 

-.0896 

NO 

WING  B 

SKILL  LEVEL  5 

TRAINED  TO  UNTRAINED 

.0615 

.8946 

NO 

TRAINED  4-  OJT  TO  UNTRAINED 

.0423 

2.9189 

YES 

*WING  B 

SKILL  LEVELS  7  &  9 

TRAINED  TO  UNTRAINED 

.1471 

-.1416 

NO* 

TRAINED  +  OJT  TO  UNTRAINED 

.1363 

-.3286 

NO* 

* WING  C 

SKILL  LEVEL  3 

TRAINED  TO  UNTRAINED 

.1921 

-1.5497 

NO* 

TRAINED  4-  OJT  TO  UNTRAINED 

.1759 

-1.5531 

NO* 

WING  C 

SKILL  LEVEL  5 

TRAINED  TO  UNTRAINED 

.0679 

1.8835 

YES 

TRAINED  +  OJT  TO  UNTRAINED 

.0643 

1.7538 

YES 

*  WING  C 

SKILL  LEVELS  7  &  9 

TRAINED  TO  UNTRAINED 

.1212 

.5239 

NO* 

TRAINED  +  OJT  TO  UNTRAINED 

.1016 

1.8172 

YES* 

* WING  A 

SKILL  LEVEL  3 

TRAINED  TO  UNTRAINED 

.1400 

-1.4084 

NO* 

TRAINED  4-  OJT  TO  UNTRAINED 

.1126 

-.4649 

NO* 

WING  A 

SKILL  LEVEL  5 

TRAINED  TO  UNTRAINED 

.0943 

-2.4840 

NO 

TRAINED  4-  OJT  TO  UNTRAINED 

.0889 

-2.5062 

NO 

*WING  A 

SKILL  LEVEL  7 

TRAINED  TO  UNTRAINED 

.3162 

-2.1082 

NO* 

TRAINED  4-  OJT  TO  UNTRAINED 

.2357 

-1.6971 

NO* 

OVERALL¬ 
SKILL  LEVEL  3 

TRAINED  TO  UNTRAINED 

.0497 

.0369 

NO 

TRAINED  +  OJT  TO  UNTRAINED 

.0416 

1.3247 

YES 

OVERALL¬ 
SKILL  LEVEL  5  i 

TRAINED  TO  UNTRAINED 

.0370 

-.8532 

NO 

TRAINED  +  OJT  TO  UNTRAINED 

.0326 

-.5394 

NO 

OVERALL¬ 
SKILL  LEVELS  7  &  9 

TRAINED  TO  UNTRAINED 

.0893 

.1620 

NO 

TRAINED  4-  OJT  TO  UNTRAINED 

.0791 

1.1130 

NO 

• 

*  Test  sample  is  not 

large  enough  (<30) .  Result  is  deleted. 

Exhibit  11-10.  QA 
STATISTICAL  RESULTS 
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Wings 

wuc 

Aircraft 

Description  of  WUC 

Wing  B;  Wing  C 

23Z00 

F-15 

F-100  Jet  Engine 

Wing  B;  Wing  C 

71AEO 

F-15 

Inertial  Measurement 
Unit 

Wing  B;  Wing  C 

74FAO 

F-15 

Radar  Transmitter 

Wing  B;  Wing  C 

74FCO 

F-15 

Radar  Receiver 

Wing  B;  Wing  C 

74FSO 

F-15 

Radar  Target  Data 
Processor 

Wing  B;  Wing  C 

74FUO 

F-15 

Radar  Set  Antenna 

Wing  B;  Wing  C 

74KAO 

F-15 

heads-up  Display  Unit 

Wing  B;  Wing  C 

74KCO 

F-15 

Signal  Data  Processor 

Wing  B;  Wing  C 

75BDO 

F-15 

Guided  Missile 

Launcher 

Wing  A 

1225A 

F— 4 

Seat  Bucket  Assembly 

Wing  A 

23Z00 

F-4 

J-79  Jet  Engine 

Wing  A 

513HO 

F-4 

Air  Data  Computer 

Wing  A 

71H00 

F-4 

Inertial  Navigation 
System 

Wing  A 

71H20 

F-4 

Navigational  Computer 

Wing  A 

71H60 

F-4 

Platform,  Gyro 
Stabilized 

Wing  A 

74B00 

F-4 

APQ-120  Radar  Set 

Wing  A 

74BBO 

F-4 

LRU-18  Control 
Oscillator 

Wing  A 

74BJO 

F-4 

Antenna  Control 

Wing  A 

74BRO 

F-4 

Wave  Guide  Assembly, 
LRU- 19 

Wing  A 

74BVO 

F-4 

LRU-16  Antenna 

Wing  A 

75130 

F-4 

Aero  7 -A  Launcher 

WUCS  CHOSEN  F 
•'ARISONS  (WUC/T 


ER-  AND 
ODOLOGY 


I 


At  this  point,  the  WUC/TA  data  was  ready  for 
comparison.  Comparisons  were  made  (inter-  and  intra-wing)  between 
work  centers  that  were  performing  the  same  action.  The  following 
figures  were  computed: 

•  average  trained  percentage  of  the  more-trained  work 
centers , 

•  average  trained  percentage  of  the  less-trained  work 
centers , 

•  average  percentage  differences  in  training,  and 

•  the  average  time  ratio. 

Exhibit  11-12  lists  these  figures  along  with  the  number  of 
comparison  groups  in  each  WUC/TA  sample.  These  overall  results 
indicate  that  increased  levels  of  training  tend  to  be  related 
to  decreased  time  to  perform  maintenance. 

2 .  Analysis 

Results  of  analyses  following  application  of  the  QA 
and  WUC/TA  skill  training  evaluation  methodologies  are  presented 
in  this  section.  We  have  included  a  discussion  of  the  biases 
that  may  affect  the  true  relationship  between  training  and 
productivity.  Each  method  is  addressed  separately. 

a .  QA  Analysis 

QA  personnel  evaluation  methodology  ovided 
inconclusive  results.  Several  valid  statistical  tests  were  made 
between  trained/passed  and  trained-plus-OJT/passed  with  untrained/ 
passed  personnel.  In  the  tests  that  compared  trained  to  untrained 


IMh- 


^Vs\Ni^Comparison  Unit 
Results 

WING  B 

1/ 

WING  C 

1/ 

WING  A 
1/ 

WING  B/C 
2/ 

(Com¬ 

bined 

Total) 

Number  of  Comparisons 

17 

119 

79 

62 

277 

Average  Trained  Per¬ 
centage  of  More-trained 
Work  Centers 

66.2% 

75.6% 

55.5% 

71.8% 

68.4% 

Average  Trained  Per¬ 
centage  of  Less-trained 
Work  Centers 

25.2% 

54.4% 

34.2% 

40.7% 

43.8% 

Average  Percentage 
Difference  in  Training 

41.0% 

21.2% 

21.3% 

31.1% 

24.6% 

Average  Time  Ratio 

.956 

1.005 

.997 

_ — — . 

.963 

j 

.  982 

/intra-wing 
/ Inter-wing 


Exhibit  11-12.  SUMMARY  RESULTS:  WUC/TA 

SKILL  TRAINING  EVALUATION 
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individuals,  18  percent  (2  out  of  11)  showed  statistically  sig¬ 
nificant  results  (i.e.,  trained  individuals  were  more  likely  to 
pass  a  QA  evaluation  than  untrained  individuals).  Testing  both 
trained  and  OJT  with  untrained  individuals  yielded  a  45  percent 
(5  out  of  11)  figure  of  statistically  significant  results.  These 
results,  however,  cannot  be  termed  as  representative  of  the 
effects  of  training  on  QA  performance  because  of  several  biases: 

•  First,  the  tasks  being  evaluated  were  not  easily 
definable  in  terms  of  what  was  actually  being  done. 
Tasks  are  often  described  in  the  QA  reports  by  Work 
Unit  Code  (WUC)  at  the  system  level,  which  is  the 
broadest  possible  description  of  a  specific  work 
action.  For  example,  23ZOO  is  a  WUC  that  describes 
work  done  on  the  F-100  engine  system.  An  operation 
that  can  be  classified  as  a  23Z00  action  can  also  be 
classified  into  a  myriad  of  subsystems  under  the  23Z00 
heading  (e.g.,  23A00-Inlet  Fan  Module) .  Similarly, 
actions  can  be  further  classified  within  subsystems  to 
the  component  level  (e.g.,  23AAC-Retaining  Plate  and 
#1  Bearing  in  the  Inlet  Fan  Module) .  Therefore,  if  is 
difficult  to  determine  the  exact  nature  of  the  task 
being  evaluated. 

•  The  relationship  between  FTD  training  received  and 
the  task  evaluated  is  hard  to  define  and,  in  some 
cases,  could  be  erroneous.  The  broad  nature  of  the 
WUC  codes  used  to  describe  work  actions  makes  it 
difficult  to  determine  what  kind  of  training  could 
be  required  for  specific  actions.  Because  of  this 
difficulty,  an  implicit  assumption  of  this  analysis 
is  that  the  FTD  training  received  by  an  individual 

is  directly  related  to  the  action  being  tested.  This 
assumption  could  be  inaccurate,  and  therefore  bias 
the  results. 

•  Although  categories  for  analysis  were  established  for 
each  wing  and  by  skill  levels,  categories  were  not  set 
up  for  specific  AFSCs .  Incorporating  more  than  one 
AFSC  into  a  testing  category  could  lead  to  a  bias  be¬ 
cause  of  the  different  nature  of  training  for  the 
different  AFSCs.  One  AFSC  could  have  several  assoc¬ 
iated  FTD  courses,  while  another  could  have  only  one. 

In  this  case,  a  person  classified  as  "trained"  in  the 
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first  AFSC  will  have  more  FTD  training  than  a  person 
"trained"  in  the  second  AFSC.  Level  of  training  (de¬ 
fined  as  the  number  of  FTD  courses  taken)  is  not  con¬ 
sidered  in  this  methodology. 

•  This  analysis  does  not  consider  how  fast  the  evaluated 
action  was  performed  because  of  the  lack  of  this  kind 
of  data  in  the  QA  reports.  This  variable,  however, 
could  be  important.  It  could  show  that  trained  indi¬ 
viduals  performed  faster  in  accomplishing  a  task.  If 
this  were  so,  our  analysis  could  be  focusing  on  the 
wrong  indicator  to  measure  the  effects  of  training. 

•  Because  of  data  limitations,  this  analysis  does  not 
consider  how  well  individuals  performed  within  the 
pass/fail  categories.  Ideally,  mistakes  are  counted 
in  the  QA  evaluation  procedure,  and  up  to  five  "minor" 
mistakes  can  be  made  before  an  individual  receives  a 
fail  rating.  Therefore,  performance  could  be  measured 
more  adequately  by  the  number  of  mistakes  made,  as 
opposed  to  the  rating  received.  In  reality,  however, 
the  number  of  mistakes  is  rarely  recorded  on  the  com¬ 
puterized  QA  reports;  the  only  measure  of  performance 
is  the  actual  rating.  Once  again,  this  analysis 
could  be  focusing  on  the  wrong  indicator  to  measure 
the  effects  of  training. 

Most  of  the  biases  that  contributed  to  the  analyses 
results  are  directly  related  to  the  database  from  which  the  QA 
methodology  was  built.  The  form  and  scope  of  available  QA  data 
is  not  particularly  suited  for  our  purposes.  Perhaps  the  most 
serious  data  biases  involve  the  lack  of  precision  in  defining 
the  task  being  evaluated  and  the  resulting  difficulty  in  linking 
FTD  training  to  specific  tasks.  A  more  in-depth  analysis  of 
the  method  by  which  specific  tasks  are  linked  to  specific  FTD 
training  courses  could  lead  to  more  conclusive  results. 

Two  tests  were  performed  on  all  categories  within 
the  QA  database; 

•  one  comparing  trained  individuals  with  untrained 
individuals,  and 
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•  one  comparing  both  trained  and  OJT  individuals  with 

untrained  individuals. 

The  rationale  for  performing  these  two  tests  for  each  category 
is  discussed  above  (Section  II.B.l.a).  Theoretically,  one  would 
expect  that  the  results  from  these  two  tests  would  be  approxi¬ 
mately  the  same,  since  FTD  course  training  and  OJT  training  serve 
the  same  purpose.  Our  results  are  inconclusive  in  establishing 
any  reliable  relationship  between  the  tests,  mainly  because 
of  data  biases.  Statistically  significant  figures  were  obtained 
for  both  methods  of  testing,  and  these  figures  seemed  to  indi¬ 
cate  that  the  trained  plus  OJT  versus  untrained  comparisons 
were  more  statistically  significant  than  the  trained  versus  un¬ 
trained  comparisons.  A  simple  analysis  of  these  statistical 
significance  figures,  however,  does  not  address  one  important 
consideration:  the  cost  versus  the  benefit  of  training  an  indi¬ 

vidual  by  the  FTD  as  opposed  to  OJT.  The  actual  time  in  training 
is  much  shorter  in  an  FTD  course.  Although  such  training  removes 
the  individual  from  the  work  center  while  the  course  is  in  ses¬ 
sion,  the  individual  returns  to  work  at  a  higher  skill  level. 
Assuming  that  an  average  time  for  OJT  training  in  an  Air  Force 
unit  is  approximately  twelve  months,  FTD  training  could  be  more 
cost  effective  in  terms  of  preparing  an  individual  to  perform 
at  a  particular  skill  level. 

b.  WUC/TA  Analysis 

The  results  of  our  WUC/TA  examination,  presented 
in  Exhibit  11-12,  generally  show  thac  increased  levels  of  training 


tend  to  be  related  to  decreased  time  to  perform  maintenance. 

These  results  could  be  erroneous,  however,  because  data-oriented 
biases  such  as  those  that  limited  our  QA  analysis  may  obscure 
more  meaningful  findings: 

•  The  task  description  for  each  WUC  is  very  general  for 
defining  similar  tasks.  Difficulties  encountered  in 
tasks  are  not  well  defined  by  such  descriptions  as 
"remove  and  replace." 

•  The  interpretation  of  personnel  awaiting  FTD  training 
as  untrained.  Actually  these  personnel  have  received 
basic  technical  training  in  their  AFSC  and,  in  some 
cases,  have  job  experience  on  other  aircraft  doing 
similar  work.  Because  of  this,  our  comparison  of  work 
centers  shows  large  differences  in  numbers  of  untrained 
personnel  and  only  small  differences  in  time  to  per¬ 
form  work.  Since  we  did  not  know  which  personnel  in 
the  work  center  did  a  specific  job,  we  assumed  all 
personnel  could  have  done  the  job.  This  may  be  com¬ 
pensated  for  by  the  fact  that  we  used  average  times 
for  several  jobs  in  the  same  work  center. 

•  Record  keeping  for  time  spent  on  a  task  may  vary  from 
one  work  center  to  another.  Although  timekeeping  is 
important,  the  requirement  to  document  all  time  while 
at  work  may  disguise  how  much  time  was  actually  spent 
on  a  particular  job. 

•  Inter-wing  comparisons  are  biased  by  different  FTD 
training,  different  approaches  to  using  personnel, 
and  different  time-keeping  procedures. 

•  The  use  of  cross-training  in  mixed  (more  than  one 
AFSC)  work  centers  may  also  lead  to  biases  in  estab¬ 
lishing  the  WUC-training  relationship  in  this  method¬ 
ology  . 

We  have  attempted  to  establish  a  relationship  between  work 
centers,  WUC,  and  AFSC  installation-level  (FTD)  training  using 
standard  output  data  from  base-level  information  systems  (MDC 
and  MMICS).  We  have  tried  to  assess  training  effectiveness 
measured  by  the  time  documented  to  do  a  task.  This  disregards 
quality  of  work  and  assumes  that  all  work  was  done  correctly. 
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Our  sample  can  be  considered  as  representative  in  terms  of  exam¬ 
ination  of  those  jobs  that  occur  with  greatest  frequency  and  use 
the  most  time  within  the  maintenance  squadrons. 

The  results  of  our  analysis,  illustrated  in  Exhibit 
11-12,  cannot  be  considered  statistically  significant.  The 
overall  weighted  variation  in  trained  personnel  was  24.6  percent, 
which  compares  with  an  overall  time  ratio  of  only  .982.  This 
can  be  partially  explained  by  the  biases  previously  listed. 

The  result  does  show,  however,  that  trained  personnel  appear 
to  have  some  effect  on  time  of  work  performance. 

3 .  Observations 

This  study  has  attempted  to  illustrate  a  verifiable, 
positive  relationship  between  FTD  training  and  job  performance. 

Two  methods  were  devised  to  illustrate  this  relationship:  a 
QA  methodology,  which  compared  individual  performance  evalua¬ 
tions  in  a  statistical  manner?  and  a  WUC/TA  methodology,  which 
compared  average  time  to  complete  a  like  task  between  work  cen¬ 
ters.  "Macro"  measures  of  performance,  such  as  the  number  of 
aircraft  hours  flown  and  the  number  of  aborted  flights  due  to 
mechanical  problems,  were  not  considered  in  this  study.  Al¬ 
though  these  measures  might  be  more  accurate  maintenance  per¬ 
formance  measures,  they  are  not  directly  relatable  to  training 
and  could  not  be  used.  The  results  of  our  work  to  date  are  not 
conclusive  although  they  indicate  that  FTD  training  has  some 
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impact  on  productivity.  The  measures  chosen  did  not  capture 
significant  differences,  but  this  is  explained  by  several  biases 
that  exist  in  the  data  that  was  used. 

This  conclusion  is  not  to  say  that  FTD  training  is 
ineffective  in  terms  of  teaching  new  job  skills.  One  must  realize 
that  personnel  who  go  through  FTD  training  have  already  received 
extensive  training  in  their  specialty  fields,  and  this  additional 
unit  training  is  a  refinement,  or  "add-on"  to  their  broad  training 
base. 

The  advantages  of  FTD  training  are  in  three  areas: 

•  capability  of  rapid  adjustment  to  local  requirements, 

•  cost  savings  based  on  little  or  no  need  to  travel 

to  distant  schools  for  training,  and 

•  rapid  return  of  students  to  the  job. 

Unit  training  goals  can  be  met  through  FTD  schooling,  and  through 
an  OJT  program.  OJT  allows  work  to  continue  without  loss  of 
students  and  instructors  to  the  local  school,  but  FTD  training 
helps  to  get  a  person  on  the  job  at  a  particular  skill  level  in 
a  shorter  period  of  time.  This  fact,  combined  with  modest  in¬ 
creases  in  productivity  (measured  either  in  quantity  or  quality 
of  work),  should  produce  higher  levels  of  aircraft  availability 
without  an  increase  in  the  size  of  the  work  force.  From  a  cost/ 
benefit  point  of  view,  any  increase  in  aircraft  availability 
yields  readiness  improvements.  This  may  be  the  most  important 
benefit  of  FTD  training. 

The  wing  commander,  in  his  attempt  to  achieve  the 
highest  number  of  mission  available  aircraft,  has  a  valuable 
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tool  in  his  QA  section.  This  group  of  highly  skilled  mainten¬ 
ance  personnel  of  varied  AFSCs  performs  an  important  function 
in  their  evaluation  of  individual  mechanics.  Of  the  2180  per¬ 
sonnel  tested  by  the  QA  sections  at  our  sample  wings  and  in¬ 
cluded  in  our  sample  data,  702  (or  32%)  failed  to  pass  the  QA 
certification.  These  personnel  had  to  be  retrained  and  then 
recertified  by  their  supervisors  that  they  were  capable  of 
performing  their  work.  Thus,  we  found  that  the  QA  section 
furnishes  a  real-time  feedback  on  the  capability  of  the  mainten¬ 
ance  personnel  to  perform  their  tasks.  Our  data  shows  that  the 
QA  program  is  viable  and  doing  its  job  of  insuring  that  mainten¬ 
ance  is  properly  performed. 
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III.  AN  EXAMINATION  OF  THE  IMPACT  OF  INSTALLATION- 
LEVEL  TRAINING  USING  SIMULATORS  ON  F-16  UNIT 
MAINTENANCE  PRODUCTIVITY 

This  section  discusses  the  potential  relationships  between 
the  use  of  Simulated  Aircraft  Maintenance  Trainers  (SAMTs)  in 
Air  Force  Field  Training  Detachment  ( FTD )  courses  and  unit  main¬ 
tenance  productivity. 

In  our  previous  research  on  FTD  training,  documented  in 
Section  II,  we  set  out  to  establish  a  linkage  between  individual 
training  conducted  at  the  installation  level  and  unit  produc¬ 
tivity.  The  results  of  that  research,  although  not  conclusive, 
did  indicate  that  FTD  training  does  have  some  impact  on  produc¬ 
tivity.  Productivity  measures  were  found  to  capture  small  dif¬ 
ferences;  biases  in  the  data  that  were  used  prevented  more  sig¬ 
nificant  results.  The  research  task  documented  in  this  section 
is  a  refinement  of  our  previous  research,  and  was  undertaken 
because  of  continued  interest  in  the  impact  of  FTD  training  on 
unit  productivity.  It  specifically  focuses  on  F-16  FTD  courses 
that  use  maintenance  simulators  in  the  curriculum,  and  the  ef¬ 
fects  that  such  training  with  simulators  has  on  maintenance  pro¬ 
ductivity.  We  did  not  quantify  the  specific  impact  of  training 
simulators . 

The  linkage  between  training  and  productivity  is  tenuous  under 
any  circumstances,  since  so  many  other  influences  impinge  on  work 
productivity.  Such  things  as  reporting  procedures,  facilities, 
and  parts  availability  can  change  the  results  achieved.  However, 
intuitively,  training  must  have  an  effect  on  maintenance  produc¬ 
tivity  and,  if  a  large  enough  sample  is  chosen  and  carefully 
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analyzed,  effects  should  become  apparent.  The  introduction 
of  a  maintenance  simulator  as  an  additional  variable  is  not  eas¬ 
ily  measured,  since  training  will  occur  whether  the  simulator  is 
available  or  not.  The  simulator  does  allow  the  student  to 
diagnose  rare  symptoms  and  thus,  on  the  occasion  that  something 
out  of  the  ordinary  occurs,  be  able  to  fix  it  properly.  The 
simulator  appears  to  provide  a  means  of  achieving  a  higher  quality 
of  maintenance,  although  our  method  of  measuring  productivity 
by  time-to-repair  will  not  make  this  readily  obvious. 

The  approach  used  in  this  task  was  to  relate  training  received 
by  maintenance  personnel  at  the  installation  level  to  on-the-job 
performance.  This  approach  included: 

•  identifying  skills  taught  using  maintenance  simulators 
at  the  wing  level,  and 

•  developing  methods  for  analyzing  the  impact  of  these 
learned  skills  on  maintenance  productivity. 

The  research  focused  on  maintenance  skills  critical  to  the 
operation  of  the  F-16  aircraft.  The  study  was  conducted  in  the 
following  order: 

•  research  the  use  of  SAMTs  by  FTDs  for  training 
maintenance  personnel  at  the  wing  level; 

•  select  appropriate  sample  organizations; 

•  request  and  receive  training  and  maintenance  productivity 
information  from  the  selected  units;  and 

•  develop  a  methodology  for  analyzing  the  data,  analyze 
the  data,  and  document  the  results. 

This  section  contains  two  subsections: 

•  Skill  Training  Evaluation,  and 

•  Application  of  Methodology  and  Results. 
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Section  III. A,  Skill  Training  Evaluation,  discusses  the  use 
of  maintenance  simulators,  analytical  considerations,  assumptions, 
sample  selection,  and  the  methodology  developed  for  analyzing 
the  relationship  between  training  and  maintenance  productivity. 

Section  III.B,  Application  of  Methodology  and  Results, 
discusses  data  and  computations,  the  application  of  our  tech¬ 
niques,  analysis  of  the  results  obtained,  and  observations. 

Supporting  information  is  presented  in  Appendix  C,  Backup 

Data . 


A.  SKILL  TRAINING  EVALUATION 

This  subsection  discusses  the  following  topics: 

•  Aircraft  Maintenance  Simulators, 

•  Analytical  Considerations, 

•  Assumptions, 

•  Sample  Selection,  and 

•  Methodology. 

1 .  Aircraft  Maintenance  Simulators 

The  use  of  aircraft  maintenance  simulators  can  be  very 
helpful  in  conducting  skill  training.  This  section  addresses 
these  topics:  the  effects  of  having  maintenance  simulators,  a 
description  of  what  kind  of  simulators  are  available  for  F-16 
training  at  the  wing  level,  and  what  the  trainers  do. 

a .  Effects  of  Having  Maintenance  Simulators 

As  a  training  device,  a  maintenance  simulator  can 
be  designed  to  provide  facilities  important  for  instructing  students, 


III-3 


4 


t  R 

a  _ 

a  “ 


;4 


ft 


t 


in  contrast  to  actual  equipment  that  is  designed  to  operate 
effectively  in  an  operational  environment.  Maintenance  simula¬ 
tors  can  be  designed  to  include  a  large  variety  of  faults  with 
which  maintenance  personnel  should  be  familiar,  including  faults 
that  cannot  be  demonstrated  conveniently  on  actual  equipment 
trainers  or  faults  that  occur  rarely  in  real  life. 

All  modern  maintenance  simulators  incorporate  some 
type  of  computer  support.  This  support  provides  numerous  train¬ 
ing  enhancements. 

•  The  computer  can  automatically  record  student  responses 
to  training  situations,  thereby  reducing  the  need  for 
constant  observation  by  the  instructor  (and  providing 
accurate  records  of  performance) . 

•  The  computer  can  assist  students  without  an  instructor's 
intervention. 

•  The  instructors  can  use  information  collected  by  the 
computer  to  better  guide  each  student  through  train¬ 
ing  by  focusing  on  weak  areas. 

Simulators  can  be  made  rugged  enough  to  sustain 
damage  or  abuse  by  students  and  thus  provide  greater  reliability 
and  availability  in  the  classroom  than  is  often  possible  with 
actual  equipment.  Training  which  would  be  avoided  because  of 
safety  reasons  (e.g.,  exposure  of  students  to  dangerous  electri¬ 
cal  charges  or  hydraulic  pressures)  can  be  undertaken  with  little 
risk  on  a  simulator.  F-16  maintenance  training  conducted  at  the 
Field  Training  Detachment  ( FTD)  level  is  enhanced  by  the  use  of 
aircraft  simulators.  These  simulators  are  organized  into  sets. 

A  full  set  may  have  11  different  simulators  or  trainers.  Each 
individual  trainer  simulates  different  functions  aboard  the 
aircraft.  These  simulators  are  divided  into  two  major  types: 
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Simulated  Aircraft  Maintenance  Trainers  (SAMTs).  These 
computerized,  fault-oriented  trainers  provide  an  ad¬ 
vanced  method  of  training  maintenance  technicians  to 
diagnose  and  correct  system  malfunctions.  SAMTs  provide 
a  method  of  teaching  operational  procedures  in  a  very 
cost-effective  manner.  Power  requirements  are  lower 
than  with  actual  equipment,  and  simulator  system  re¬ 
sponse  to  student  inputs  is  identical  to  that  of  the 
aircraft. 

Hardware  Trainers.  These  trainers  provide  a  physical 
simulation  of  actual  maintenance  areas.  These  simu¬ 
lators  focus  on  "hands-on"  training  experiences  for 
the  repair  of  common  malfunctions  in  some  of  the  more 
mechanical  areas  of  the  aircraft;  diagnostic  skills 
are  not  stressed  within  these  simulators. 


b.  F-16  Aircraft  Maintenance  Simulators 


The  full  group  of  simulators  is  called  a  mobile 
training  set  (MTS)  and  consists  of  seven  SAMTs  and  four  hardware 
simulators  (or  trainers).  Each  simulator  is  numbered,  although 
not  all  numbers  are  used.  Six  SAMTs  are  manufactured  by  Honeywell 
and  one  by  Educational  Computer  Corporation  (ECC).  The  hardware 
trainers  are  made  by  General  Dynamics.  The  list  below  shows 
simulator  number,  title  and  manufacturer; 


Simulator 

Number 


Simulator  Title  and  Manufacturer 


SAMT — Flight  Control/lnstrumentation  (Honeywell) 
SAMT — Navigation  (Honeywell) 

SAMT — Electronics  (Honeywell) 

Hardware--Seat  and  Canopy  (General  Dynamics) 

SAMT — Hydraulics  (Honeywell) 

SAMT — Engine  Start  (Honeywell) 

SAMT--Engine  Diagnostics  (Honeywell) 

SAMT--Engine  Operating  Procedure  (Honeywell) 
Hardware--F-1Q0  Engine  (General  Dynamics) 
Hardware--Gun  (General  Dynamics) 

Hardware— Fuel  (General  Dynamics) 

SAMT — Environmental  Control  (ECC) 


*No  longer  in  use. 
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The  following  is  a  listing  of  where  MTSs  have  been 
provided  for  training  support  in  the  Continental  U.S.  (CONUS). 


Location  Supported  Unit 


MTS  Status  (Issue  Dates) 


Hill  AFB 


388th  TFW 


Issued  MTS  1  (Jan.  -  Dec.  79) 


MacDill  AFB 


56th  TTW 


Issued  MTS  2  (Feb.  79  -  May  80) 
MTS  2  moved  to  Shaw  AFB  in  Jan. 
Issued  MTS  5  (Apr.  83). 


Shaw  AFB 


363rd  TTW 


Issued  MTS  2  (Jan.  82) 


Luke  AFB 


58th  TTW 


Issued  MTS  6  (Aug.  -  Sep.  82) 


Nellis  AFB 


474th  TFW 


Issued  MTS  4  (Nov.  82  -  Apr.  83) 


This  information  is  shown  graphically  on  Exhibit  III-l 


MOV  APR  ... 


NELLIS  AFB  A" - 


AUG-SEP 


LUKE  AFB 


SHAW  AFB 


MACDILL  AFB  A" 


HILL  AFB  A" 


^ISSUED 


Exhibit  III-l.  MTS  LOCATIONS  (CONUS) 
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Use  of  Simulators 


c . 

The  following  listing  shows  the  simulator 
number  and  title,  aircraft  component  repaired  by  primary  work 
unit  codes  (WUCs)  that  are  part  of  the  malfunctions  taught  using 
the  simulators,  and  the  course  title/AFSC  for  which  the  simulator 
is  used. 


Aircraft 

Equipment  Repaired-WUCs  Courses/AFSC 


Simulator 

2,  Flight  Control/ 
Instrumentation 


14A00,  Primary  Flight 
Control  Electronics 
14B00,  Primary  Flight 
Control  Actuators 
51A00,  Primary  Flight 
Instruments 
51F00,  Control  Air 
Data  Computer 


Integrated  Avionics 
Instrument  and  Flight 
Control  System 
Specialist  (F-16); 
for  AFSC  326X7 


3,  Navigation 


71A00,  TACAN  Naviga¬ 
tion  Set 

71B00,  Instrument 
Landing  Set 


Integrated  Avionics, 
Navigation,  and  Pen¬ 
etration  Aids  Systems 
Specialist  (F-16);  for 
AFSC  326X8. 


4,  Electronics 


42000,  Electrical  Aircraft  Electrical 

Power  Supply  Systems  Technician 

(F-16);  for  AFSC  423X0 


6,  Seat  and  Canopy  12000,  Crew  Station 

System 


Aircrew  Egress  Systems 
Technician  (F-16);  for 
AFSC  423X2. 


10,  Engine  Start 

I  23000,  Turbofan 

11,  Engine  Diagnos- (  Power  Plant;iary 

tics  }  24000,  Auxiliary 

12,  Engine  Operat-  t  Power  Plant 

ing  Procedure  \ 

13,  F-100  Engine  I 


Jet  Engine  Technician 
(F-16);  for  AFSC  426X2 
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Simulator 


14,  Gun 


Aircraft 

Equipment  Repaired-WUCs 
75AOO,  Gun  System 


Courses/AFSC  (Cont'd) 

Weapons  System  Maint¬ 
enance  Technician 
( F— 1 6 ) ;  for  AF SC  462X0. 


15,  Fuel 


46000,  Fuel  System 


Aircraft  Fuel  Systems 
Technician  (F-16); 
for  AFSC  423X3. 


22,  Environmental  41000,  Environmental 

Control  Control  System 


Aircraft  Environmen¬ 
tal  System  Technician 
(F-16);  for  AFSC  423X1 


2 .  Analytical  Considerations 

This  study  examines  the  impact  of  FTD  training, 
particularly  the  training  that  incorporates  the  use  of  mainten¬ 
ance  simulators,  on  maintenance  productivity.  In  our  previous 
research,  we  found  that  the  Air  Force  Quality  Assurance  (QA) 
program  worked  well  and  had  an  impact  on  the  care  taken  in  air¬ 
craft  maintenance  as  well  as  the  assurance  that  maintenance  per¬ 
sonnel  were  maintaining  aircraft  properly  and  safely.  however, 
no  quantitative  measures  were  developed  from  that  particular 
research  that  could  be  used  to  demonstrate  the  linkage  between 
training  and  maintenance.  Our  previous  work  did  show  that  the 
use  of  work  unit  code  trend  analyses  (which  focus  on  time  to 
complete  a  maintenance  action)  might  prove  to  be  a  useful  tool 
in  linking  training  and  maintenance  productivity.  Therefore, 
in  this  phase  of  our  work  we  have  focused  on  that  aspect  --  time 
to  complete  a  maintenance  action  --  as  the  measure  of  performance 
and  related  it  to  training  conducted  at  the  unit  (wing)  level. 
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Many  factors  beyond  training  at  the  unit/base  level 
and  the  use  of  actual  or  simulated  equipment  can  profoundly  in¬ 
fluence  our  capability  to  maintain  military  equipment.  These 
factors  include: 

•  the  quality  of  personnel  recruited  by  the  military 
services , 

•  the  policies  used  by  the  Services  to  assign  recruits 
to  occupational  specialties, 

•  the  amount  and  type  of  training  to  be  accomplished 
at  technical  schools, 

•  the  complexity  of  the  information  that  must  be 
assimilated  in  order  to  accomplish  maintenance, 

•  quality  of  maintenance  supervision, 

•  equipment  design,  and 

•  maintenance  policy. 

These  factors  are  recognized  as  being  important 
considerations  in  equipment  maintenance  analyses,  but,  due  to 
our  specific  concern  in  this  study  on  training  and  maintenance 
simulators,  these  factors  are  not  explicitly  addressed. 

3 .  Assumptions 

The  analytical  task  of  linking  training,  frequency, 
and  productivity  measures  is  based  on  several  major  assumptions 
relating  to  the  data  and  the  procedures  followed  to  process  the 
data.  These  assumptions  include  the  following. 

•  An  implicit  assumption  in  this  analysis  is  that  the  data 
being  analyzed  is  accurate  and  complete.  Problems  with 
data  recording  and  data  entry  could  lead  to  biased  or 
inaccurate  data  analyses.  In  spite  of  these  possible 
inaccuracies  in  the  data  collection  system,  this  study 
is  based  on  information  inputted  to  the  MDC  system. 

Data  which  is  verifiable  as  inaccurate  is  excluded  from 
our  analysis.  All  other  data  is  included. 
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•  Data  entered  in  the  maintenance  data  collection  ( MDC ) 
system  is  assumed  to  represent  "successfully  completed" 
work  actions. 

•  Time  of  performance  shown  on  MDC  printouts  for  work 
actions  is  the  elapsed  time  (difference  between  start 
and  stop  time). 

•  We  are  assuming  that  the  current  work  center  training 
percentages  are  indicative  of  the  training  percentage 
of  each  work  center  over  time.  Training  data  is  only 
available  for  the  current  time  period,  whereas  the  pro¬ 
ductivity  data  that  is  associated  with  this  training 
data  is  obtained  for  a  period  of  six  months.  The  number 
of  trained  personnel  or  "training  percentage"  that  we 
use  for  a  work  center  can  change  drastically  over  that 
period  of  time;  however,  because  of  the  lack  of  more 
complete  data,  the  current  training  percentage  is  used 
in  our  analysis  for  all  work  centers. 

•  Each  member  of  a  work  center  has  an  equal  opportunity 
to  perform  a  particular  action.  Thus,  for  all  work 
performed,  the  calculations  will  use  the  total  work¬ 
ers  (with  appropriate  AFSC)  in  each  work  center. 

•  We  can  identify  particular  WUCs  that  are  associated 
with  actions  taught  using  maintenance  simulators. 
However,  these  WUCs  are  often  very  specific.  In  order 
to  match  this  information  with  the  level  of  detail 
obtained  in  our  WUC-related  data  requests,  we  are 
assuming  that  associated  higher  level  WUCs  are  also 
taught,  in  sufficient  detail,  on  the  SAMTs . 

•  If  a  maintenance  simulator  is  supposed  to  be  used  in 
training,  and  an  MTS  is  present  at  a  base,  then  we 
are  assuming  that  the  SAMT  is  used  in  all  appropriate 
training  (we  are  not  accounting  for  any  simulator  down¬ 
time  ) . 


4.  Sample  Selection 


For  this  task,  we  chose  a  sample  large  enough  to  assure 
reasonable  results  for  our  analysis.  We  chose  the  two  F-16  wings 
which  had  used  simulators  in  their  FTD  training  for  a  period  of 
over  one  year.  We  also  chose  another  F-16  wing  which  had  never 
had  simulators  for  their  FTD  training  and  had  the  F-16  for  a  long 
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period  of  time  (over  three  years)  so  comparative  analyses  might 
be  made.  Our  final  selection  of  sample  wings  was  as  follows: 

•  Hill  AFB,  UT  —  388th  TFW, 

•  MacDill  AFB,  FL  —  56th  TTW,  and 

•  Nellis  AFB,  NV  —  474th  TFW. 

An  examination  of  Exhibit  III-l  shows  that  at  Hill  AFB, 
all  training  used  simulators,  all  Nellis  AFB  training  did  not 
use  simulators,  and  most  MacDill  AFB  training  used  simulators. 

The  identity  of  the  wings  we  examined  is  not  relevant; 
hence,  we  have  randomly  titled  them  as  Wing  J,  Wing  K,  and  Wing  L 
The  work  centers  for  the  wings  were  assigned  numbers  for 
identification.  This  alpha-numeric  code  is  used  throughout 
the  analysis  portion  of  this  section  to  identify  specific  work 
centers. 

5 .  Methodology 

The  methodology  used  for  examining  the  linkage  between 
FTD-provided  training,  specifically  that  training  using  simu¬ 
lated  aircraft  maintenance  trainers  and  hardware  trainers,  and 
maintenance  productivity  relies  heavily  on  the  work  MCR  accom¬ 
plished  in  our  previous  research.  The  inference,  or  intuitive 
sensing,  is  that  as  the  number  of  FTD-trained  personnel  in  a 
work  center  increases,  the  productivity  of  that  work  center  will 
increase.  Productivity  is  measured  in  terms  of  the  worker  hours 
used  to  complete  a  specific  work  action.  Obviously,  the  complex¬ 
ity  of  work  actions  can  vary  even  when  a  specific  subassembly  is 
being  worked  on  and  the  action  code  used  is  fairly  specific.  Thi: 
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complexity  or  difference  in  work  performed  should  be  statisti¬ 
cally  smoothed  by  taking  a  large  enough  sample  of  particular 
types  of  work. 

The  following  subsections  will  address  three  subjects: 

•  Determination  of  Training  Status, 

•  Determination  of  Maintenance  Productivity,  and 

•  Analytical  Techniques. 


a.  Determination  of  Training  Status 

The  process  of  determining  the  training  status  of 
specific  work  centers  involves  obtaining  the  training  require¬ 
ments  for  the  skills  involved  in  performing  the  work,  finding 
which  individuals  have  received  the  required  FTD  course  training, 
and  calculating  the  overall  work  center  training  status.  The 
procedure  we  followed  in  the  wings  examined  in  this  task  for 
calculating  trained/untrained  status  is  listed  below. 

•  Determine  which  skills  might  be  examined  based  on  the 
maintenance  simulators  provided  by  the  Air  Force  for 
FTD  training. 

•  Obtain  copies  of  programs  of  instruction  (POIs)  published 
by  Air  Training  Command  for  these  skills.  Examine 

the  POIs  to  see  which  maintenance  simulators  are  used 
in  FTD  courses  for  the  appropriate  skills.  This  ex¬ 
amination  results  in  a  determination  of  whether  the 
training  uses  the  maintenance  trainer  in  a  manner  that 
appears  to  assure  a  measurable  impact  on  task  perfor¬ 
mance  for  a  specific  skill. 

•  Request  course  status  summaries  from  each  wing  for  the 
skills  chosen  for  examination. 

•  Calculate  trained/untrained  personnel  status  for  any 
work  center  that  will  be  examined.  The  examination 
of  specific  work  centers  is  determined  by  analysis 
of  maintenance  productivity  information.  Only  those 
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work  centers  that  had  a  measurable  sample  of  work 
actions  were  chosen-  This  is  explained  in  Section 
III. B. 1 . 

•  The  determination  of  whether  a  person  is  trained  or 
untrained  proceeds  as  follows: 

If  a  person  has  taken  the  required  FTD  course 
for  his  AFSC ,  he  is  considered  to  be  trained. 

-  If  a  person  has  not  taken  the  required  FTD  course 
for  his  AFSC,  he  is  considered  to  be  untrained . 

•  Calculate  work  center  trained  percentages  for  specific 
skills  (AFSCs ) .  When  a  work  center  contains  only  one 
AFSC,  then  the  calculation  consists  of  dividing  the 
number  of  trained  personnel  by  the  total  number  of 
personnel  in  the  work  center.  When  a  work  center  has 
more  than  one  AFSC,  then  the  calculation  consists  of 
dividing  the  number  of  trained  personnel  in  the  AFSC 
that  pertains  to  the  work  performed  by  the  total  num¬ 
ber  of  personnel  having  that  particular  AFSC  in  the 
work  center. 

•  Separate  work  centers  by  training  level.  Training 
level  is  defined  as  the  percentage  of  trained  per¬ 
sonnel  in  the  work  center  for  a  specific  AFSC. 


b.  Determination  of  Maintenance  Productivity 

The  procedure  for  determining  the  maintenance 
productivity  of  specific  work  centers  starts  with  the  choice 
of  specific  work  unit  codes  (WUCs)  to  be  examined.  This 
choice  was  assisted  considerably  by  the  help  of  Honeywell,  Inc. 
personnel  who  provided  information  on  each  maintenance  simulator 
they  make  and  the  specific  WUCs  that  each  particular  simulator 
represents.  This  data  is  attached  in  Appendix  C. 

Our  primary  interest  was  on  the  SAMTs,  so  initially 
we  chose  to  request  information  only  for  WUCs  associated  with 
them.  However,  WUC  12000  was  also  chosen  so  we  could  examine 
data  for  a  hardware  trainer  (Crew  Station  System  —  canopy  and 
seat ) . 
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Previously,  in  our  research  on  the  F-15/F-4  aircraft, 
we  had  used  a  standard  output  from  the  Air  Force  Maintenance  Data 
Collection  (MDC)  system  known  as  the  WUC  trend  analysis  report. 
However,  a  new  system  known  as  the  Consolidated  Data  System  (CDS) 
has  been  instituted  for  the  F-16  that  allows  for  more  flexible 
and  responsive  maintenance  data  reporting.  Both  systems  rely  on 
maintenance  information  recorded  by  each  work  center  on  Air  Force 
Form  AFTO  349,  "Maintenance  Data  Collection  Record, "  which  is 
entered  in  the  MDC  data  base  at  each  wing.  CDS  draws  off  the 
MDC  data  base  to  display  whatever  information  is  required.  The 
advantage  of  CDS  over  MDC  is  the  ability  to  aggregate  maintenance 
data  in  a  more  usable  form  and  with  flexibility  as  to  which  in¬ 
formation  is  displayed.  The  obvious  information  needed  is  what 
was  worked  on  (WUC),  what  was  done  (action  taken  code),  how  long 
the  action  took  (time),  and  who  performed  the  action  (work  center). 

After  determining  which  WUCs  are  to  be  examined, 
a  request  to  each  wing  for  a  specifically  formatted  report  for  a 
particular  time  period  resulted  in  a  large  amount  of  information 
for  use  in  analysis.  We  were  assisted  greatly  in  our  data  request 
and  formats  by  representatives  of  the  Tactical  Air  Command  (TAC), 
located  at  their  headquarters,  as  well  as  TAC  personnel  at  the 
wing  level  who  had  been  working  with  the  CDS. 

Our  general  procedure  for  determining  work  center 
maintenance  productivity  using  the  output  data  from  the  CDS  is 
as  follows. 
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•  Request  a  WUC  report  that  specifies  the  WUCs  required 
for  analysis.  The  time  period  requested  was  calendar 
year  1982.  The  report  was  requested  for  all  work 
centers  and  for  all  action  taken  codes. 

•  After  receipt  of  the  WUC  report  from  each  wing,  examine 
it  for  completeness.  Then  see  which  work  centers  have 
the  appropriate  AFSCs  for  the  WUCs  under  consideration. 

•  Determine  if  sufficient  WUC  actions  occur  for  a  part¬ 
icular  work  center  (to  include  action  codes)  to  indi¬ 
cate  a  normal  distribution  (i.e.,  30  or  more  actions). 

•  Separate  and  aggregate  the  WUCs  by  work  center  and 
action  taken  code.  Then  sum  to  get  total  hours  for 

each  work  center/action  taken  code.  At  this  point, 
the  analysis  of  maintenance  productivity  must  incor¬ 
porate  other  variables;  i.e.,  total  number  of  actions 
taken  by  each  worker  (frequency)  and  level  of  work 
center  training.  Number  of  workers  and  training  in¬ 
formation  is  received  from  other  wing  data  bases. 

•  The  final  analytical  step  is  to  make  comparisons 
among  work  centers  to  see  what  effects  training  and 
other  measurable  variables,  in  this  instance  fre¬ 
quency,  have  on  maintenance  productivity. 

c.  Analytical  Techniques 

The  term  analytical  technique  means  the  integrated 
step-by-step  process  followed  in  analyzing  the  training  and 
maintenance  action  data  that  we  requested/received  from  three 
Air  Force  F-16  wings. 

We  used  the  following  systems  as  our  data  sources: 

•  the  Consolidated  Data  System  (CDS)  for  F-16  maintenance 
which  provided  maintenance  information,  and 

•  the  training  subsystem  of  the  Maintenance  Management 
Information  and  Control  System  (MMICS),  which  provided 
training  information. 

Special  terms  used  in  our  analysis  are  defined  below. 

•  Observation  is  a  single  completed  work  action. 
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•  Frequency  is  the  total  nu  'ber  of  observations  divided 
by  the  total  number  of  w.  ers  that  did  the  work: 

frequency  =  number  of  observations  (or  work  actions) 

number  of  workers 


Productivity  is  total  elapsed  time  (in  hours ) .divided 
by  frequency  or  worker-hours  per  observation.— 


productivity  =  total  elapsed  time 

frequency 


(number  of  workers )x (elapsed  time) 
number  of  work  actions 

It  is  important  to  note  that  as  the  frequency  figure 
decreases,  productivity  is  increasing,  since  the  mea¬ 
surement  is  worker-hours  per  observation  and  fewer 
hours  spent  on  a  work  action  is  an  increase  in  produc¬ 
tivity. 

We  used  three  approaches  to  examine  the  data.  These 


approaches  were: 


•  examine  productivity  by  action  code, 

•  examine  productivity  by  frequency,  and 

•  use  analysis  of  variance  (ANOVA)  techniques. 


Each  of  these  is  discussed  below. 


( 1 )  Examination  of  Productivity  by  Action  Code 
This  approach  involves: 

•  aggregating  the  elapsed  time  for  a  specific  WUC  by 
action  taken  code  ( "action  taken" )  for  each  work 
center, 

•  dividing  the  total  elapsed  time  by  the  frequency  to 
get  productivity. 


4/We  chose  the  standard  industrial  engineering  definition  for 
productivity  (i.e.,  lower  is  better  since  it  is  time  required 
to  perform  a  task  or  produce  an  output)  rather  than  the  sometimes- 
used  alternate  definition  (i.e.,  output  per  unit  of  time). 
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•  consolidating  productivity  data  by  wing  and  by  training 
level, 

•  plotting  the  resulting  productivity  data  by  individual 
action  code,  and 

•  examining  the  result  considering  training  levels. 

( 2 )  Examination  of  Productivity  by  Frequency 
This  approach  involves  exactly  the  same  steps 

listed  above  in  "Examine  Productivity  by  Action  Code"  except  the 
productivity  data  is  plotted  by  frequency  per  action  code  and 
training  is  not  considered  in  the  examination.  We  also  fitted  a 
regression  line  to  the  data  to  depict  the  impact  on  productivity. 

The  slope  of  the  regression  denotes  increasing  or  decreasing  pro¬ 
ductivity  (negative  slope  —  increasing). 

( 3 )  Use  of  ANOVA  Techniques 

This  approach  involves  a  statistical  test  of 
sample  results  to  determine  whether  or  not  the  results  observed 
are  statistically  significant.  The  steps  in  this  technique  include: 

•  classification  of  all  work  center  observations  into 
"cells"  in  an  ANOVA  matrix; 

•  computation  of  average  productivity  figures  within 
each  ANOVA  matrix; 

•  computation  of  variation  statistics  attributable  to 

training, 

-  frequency,  and 

the  error  term;  and 

•  performing  an  F-test  for  variable  significance. 


III-17 


3.  APPLICATION  OF  METHODOLOGY  AND  RESULTS 

We  used  three  analytical  techniques  during  this  phase.  The 
first  two  are  quite  basic  (productivity  by  action  code  and  produc 
tivity  by  frequency) .  Statistical  methods  were  used  to  compute 
the  regression  line  for  the  second  procedure  and  for  the  entire 
ANOVA  procedure.  This  section  discusses  the  following: 

•  Data  and  computations, 

•  Examination  of  Productivity  by  Action  Code, 

•  Examination  of  Productivity  by  Frequency, 

•  Use  of  ANOVA  Technique,  and 

•  Observations. 


1 .  Data  and  Computations 


Before  discussing  our  analytical  techniques,  we  will 
explain  the  steps  which  were  used  in  the  selection  of  the  WUCs 
and  action  taken  codes  for  our  analysis.  In  section  III.A.l.c, 
we  displayed  10  WUCs  for  the  11  simulators  which  make  up  an  MTS. 
These  WUCs  are  listed  below. 


NUMBER 


NAME 


12000 

14000 

23000 

24000 

41000 

42000 

46000 

51000 

71000 

75000 


Crew  Station  System 
Flight  Control 
Turbofan  Power  Plant 
Auxiliary  Power  Plant/ Jet  Fuel 
Starter 

Environmental  Control  System 
Electrical  Power  Supply 
Fuel  System 
Flight  Instruments 
Radio  Navigation 
Weapons  Delivery 


1 1 1  - 1 8 


First,  we  chose  those  WUCs  associated  with  Honeywell 
simulators  since  we  had  detailed  information  on  them.  We  added 
one  WUC  associated  with  a  hardware  simulator. 

•  Eliminate:  41000,  46000,  and  75000 

We  then  requested  workload  data  on  the  remaining  seven  WUCs  from 
three  wings. 

After  receipt  of  the  workload  data  from  the  wings,  our 
second  step  was  to  examine  the  documents  to  find  which  WUCs  had  a 
sufficient  number  of  observations  for  our  initial  analysis.  One 
WUC  was  eliminated. 

•  Eliminate:  24000 

Based  on  the  preceding  examination,  we  eliminated  several 
action  taken  codes  and  subordinate-level  WUCs .  The  remaining 
WUCs  and  action  taken  codes  were  then  aggregated  for  further 
examination.  These  WUCs  were:  12CA0,  12E00,  14A00,  14B00, 

23ZOO,  42A00,  42B00,  42C00,  51A00  and  51F00. 

In  the  last  step,  we  were  able  to  accurately  determine 
which  WUCs  and  associated  action  taken  codes  had  sufficient 
numbers  of  observations  for  analysis.  This  determination  gave  us 
WUC  14A00  with  six  action  taken  codes  and  WUC  23Z00  with  three 
action  taken  codes.  The  WUCs  we  chose  make  up  approximately  5% 
of  total  wing  maintenance  and  represent  a  significant  part  of  the 
total.  As  a  result  of  our  selection  process,  the  following  action 
codes  were  chosen: 
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14A00 


23Z00 


TITLE 


P  P  Removed 

Q  Installed 

R  Removed  and  Replaced 

T  Removed  for  Cannibalization 

U  Replaced  after  Cannibalization 

X  X  Test-Inspection-Service 

Y  Troubleshoot 

We  limited  the  number  of  WUCs  which  were  investigated 
to  those  taught  using  simulators.  This  allowed  us  to  establish 
a  close  relationship  between  maintenance  training  conducted  using 
simulators  and  the  work  performed  on  the  job.  There  are  addi¬ 
tional  WUCs  which  are  being  worked  on  by  the  people  in  the  work 
centers  with  the  same  AFSC.  An  analysis  of  this  worker-time  re¬ 
lationship  showed  that  the  WUCs  we  examined  did  not  constitute 
the  entire  work  accomplished  by  those  specific  work  centers. 

This  could  bias  the  frequency  value  included  in  the  present 
analyses  because  the  number  of  people  wholly  devoted  to  the  ob¬ 
servations  we  examined  is  less  than  the  total  number  of  people 
in  the  work  center.  Our  calculations  assumed  that  all  workers 
(with  appropriate  AFSCs )  worked  on  the  observations  we  examined. 
The  computations  of  the  frequencies  for  the  two  WUCs  we  looked 
at  are  contained  in  Appendix  C. 

The  number  of  observations  per  worker,  or  frequency, 
was  explained  previously.  The  productivity  figure  we  used  for 
worker  hours  per  observation  allows  for  comparison  of  work  cen¬ 
ters  of  different  sizes.  As  opposed  to  our  frequency  figure, 
where  the  higher  the  numeric  value  the  better  the  result,  our 
productivity  figure  exhibits  the  opposite  characteristic  --  the 
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lower  the  numeric  value  the  better  the  result  (lower  worKer-hours 
per  observation  is  "better").  The  productivity  computations  for 
the  two  WUCs  are  contained  in  Appendix  C . 

In  order  to  assure  that  the  amount  of  time  spent  by 
work  centers  for  the  actions  we  examined  was  representative,  we 
did  a  comparison  of  hours  worked  in  WUC  14A00  and  WUC  23Z00  to 
the  total  hours  of  work  completed.  There  are  28  system  level 
WUCs  for  the  F-16.  WUC  14000  was  the  highest  manhour  consumer 
in  our  sample  (10.6%),  WUC  23000  was  fourth  with  8.0%.  There 
are  seven  subsystems  within  WUC  14000 — WUC  14A00  was  32%  of 
WUC  14000  or  3.4%  of  total  wing  manhours.  There  are  12  subsys¬ 
tems  within  WUC  23000 — WUC  23Z00  was  24%  of  WUC  23000  or  2.0% 
of  total  wing  manhours.  Thus  out  of  113  subsystem  WUCs,  the 
two  WUCs  we  examined  (14A00  and  23Z00)  are  quite  representative 
of  total  wing  maintenance  since  they  consume  over  5%  of  total 
maintenance  manhours  in  the  sample  we  looked  at. 

2 .  Examination  of  Productivity  by  Action  Code 

This  section  discusses  the  productivity  by  action 
code  technique.  It  is  divided  into  two  parts: 

•  Application,  and 

•  Results. 


a .  Application 

The  examination  of  productivity  by  action  code 
required  the  calculation  of  productivity  data  by  work  center. 
These  calculations  are  provided  in  Appendix  C.  The  application 
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also  required  the  training  status  of  the  individual  work  centers 
examined.  The  training  status  (percent  trained)  by  work  center, 
as  well  as  the  number  of  observations  for  each  action  code,  are 
shown  for  the  two  WUCs  we  examined  on  Exhibits  III-2  and  III-3. 

Using  the  computed  work  center  and  wing  productivity, 
we  plotted  the  data  by  action  code  for  WUCs  14A00  and  23Z00.  These 
exhibits  (III-4  through  III-10)  also  show  the  training  and  fre¬ 
quency  information  pertinent  to  each  plot.  The  lines  connecting 
the  plotted  points  are  strictly  for  interpretive  purposes,  such 
as  noting  trends  or  changes.  Because  three  variables  (produc¬ 
tivity,  training,  and  frequency)  are  all  shown  on  the  graphs,  the 
frequency  for  each  action  code  is  given  along  with  the  percent 
of  workers  trained  for  each  organizational  entity.  The  charts 
are  explained  below. 

•  Wing-to-wing  (Exhibits  III-4  and  III-5).  These  exhibits 
show  any  trends  that  may  exist  between  bases,  based  on 
the  percent  of  workers  trained.  At  the  same  time,  one 
can  also  see  the  values  of  the  frequencies  for  each  of 
the  data  points.  In  the  ANOVA  section,  these  two  var¬ 
iables  will  be  isolated  and  their  impact  discussed. 

•  High,  medium,  and  low  training  (Exhibits  III-6,  -7, 

-8,  -9,  -10).  These  five  exhibits  each  deal  with  one 
category  of  training,  based  upon  the  percent  of  trained 
individuals  within  each  work  center.  Shown  below  is 

a  display  of  the  number  of  work  centers  for  each  WUC 
that  falls  into  one  of  the  three  categories. 
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Exhibit  III-2 .  TRAINING  STATUS  AND  NUMBER  OF  OBSERVATIONS  (14A00) 


(%  TRAINED) 


Exhibit  III-4 .  TRAINING:  WING-TO-WING  (14A00) 


Exhibit  I I 1-5 .  TRAINING:  WING-TO-WING  (23ZOO) 


(%  TRAINED) 
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Exhibit  III-7.  TRAINING:  MEDIUM  (14A00) 


(X  TRAINED) 
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(%  TRAINED) 


Exhibit  III-10.  TRAINING:  MEDIUM  (23ZOO) 
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b.  Results 


The  graph  on  Exhibit  III-4  allows  us  to  make  a 
comparison  of  the  effect  training  has  on  productivity  for  WUC 
14A00.  We  see  examples  of  varying  levels  of  training  for 
the  work  centers  at  each  of  the  three  wings .  The  graph  on  Exhibit 
III-5  for  WUC  23Z00  does  not  allow  us  to  make  as  good  a  com¬ 
parison  since  two  of  the  wings  have  the  same  trained  percentage 
(74%).  However,  both  of  these  wings  did  significantly  better 
than  the  wing  with  the  lower  trained  percentage  of  59%. 

Our  results  are  further  discussed  by  WUC  (14A00 
and  23Z00)  below. 

•  WUC  14A00 

Wing-to-wing  (Exhibit  III-4).  This  chart  shows 
what  would  normally  be  expected  —  the  greater 
the  degree  of  worker  training,  the  better  the 
productivity . 

High  training  (Exhibit  IIX-6) .  Because  there  is 
only  one  work  center  in  the  high  training  category, 
no  comparison  can  be  made  with  other  work  centers 

Medium  training  (Exhibit  III-7).  All  eight  work 
centers  are  relatively  close  in  the  percent  of 
their  personnel  trained  (36-55%),  and  follow  a 
fairly  consistent  pattern  with  regard  to  pro¬ 
ductivity.  In  other  words,  the  most  productive 
work  centers  generally  remain  the  most  productive 
for  all  action  codes. 

In  this  category,  frequency  does  not  appear 
to  play  a  significant  role.  The  eight  work  centers 
do  not  appear  to  follow  any  consistent  pattern 
between  high  frequency  and  productivity.  Therefore, 
in  our  medium  training  comparison,  neither  training 
nor  frequency  appear  to  have  a  relationship  to  pro¬ 
ductivity  . 


II 1-32 


> 


Low  training  (Exhibit  III-8).  This  comparison 
includes  only  two  work  centers,  and  does  not 
exhibit  any  consistent  patterns  with  respect  to 
training  or  frequency  and  productivity.  The  bet¬ 
ter  trained  (20%  vs.  11%)  work  center  is  more 
productive  in  three  of  the  six  action  codes.  The 
work  center  with  the  highest  frequency  is  also 
only  the  most  productive  in  three  of  the  six 
action  codes.  It  appears  that,  when  a  work  cen¬ 
ter  is  essentially  "untrained, "  being  most  pro¬ 
ductive  is  a  hit-or-miss  affair. 

-  In  summary,  although  there  are  occasional  con¬ 
sistent  trends  with  regard  to  either  training 
or  frequency  for  a  particular  level  of  training, 
these  trends  do  not  hold  for  all  the  analyses. 

•  WUC  23Z00 

Wing-to-wing  (Exhibit  III-5).  In  this  comparison, 
training  appears  to  bear  some  relationship  to  pro¬ 
ductivity.  The  two  highest  trained  bases,  at  74%, 
were  both  more  productive  for  each  action  code  than 
the  least  trained  base,  at  59%.  However,  frequen¬ 
cy  does  not  appear  to  have  a  consistent  relation¬ 
ship  to  productivity.  In  two  of  the  three  action 
codes,  the  least  productive  wing  has  higher  fre¬ 
quencies  than  the  two  more  productive.  In  compar¬ 
ing  the  two  most  productive  bases  in  all  three 
action  codes,  the  base  with  the  highest  composite 
frequency  is  also  the  most  productive.  Thus,  for 
this  comparison,  training  may  have  an  influence 
while  frequency  appears  to  have  little  consistent 
pattern. 

-  High  training  (Exhibit  III-9).  For  WUC  23Z00, 
eight  of  the  eleven  work  centers  are  in  this  cat¬ 
egory.  Their  training  percentages  are  closely 
grouped  between  70%-78%  --  essentially  all  "equally 
trained."  Although  there  is  no  absolute  pattern 
between  work  centers  with  regard  to  their  produc¬ 
tivity  standings,  there  are  some  fairly  consistent 
groupings.  With  one  exception,  J12,  the  five 

most  productive  work  centers  remained  so  for  all 
action  codes.  The  three  least  productive  remained 
the  least  product i  /e.  There  was  no  consistent 
pattern  with  regard  to  frequency.  For  a  particu¬ 
lar  action  code,  the  work  centers  with  the  highest 
frequency  were  the  least  productive.  Conversely, 
work  centers  with  very  low  frequencies  were  the 
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most  productive.  All  in  all,  little  can  be  learn¬ 
ed  from  this  comparison  regarding  a  consistent 
relationship  between  productivity,  training,  and 
frequency • 

Medium  training  (Exhibit  III-10) .  The  remaining 
three  work  centers  are  in  this  category  with 
training  percentages  of  42%,  54%,  and  56%.  As  in 
the  high  training  category,  no  consistent  pattern 
emerges  between  training  and  productivity.  The 
least  trained  work  center  is  the  most  productive 
in  two  of  the  three  action  codes.  The  highest 
trained  work  center  is  the  least  productive  in 
two  of  the  three  action  codes.  Frequency  also 
appears  to  exhibit  no  consistent  pattern.  The 
work  center  with  the  highest  frequency  is  in  the 
middle  for  productivity.  The  work  center  with 
the  lowest  frequency  is  the  least  productive 
for  two  of  the  three  action  codes  and  the  most 
productive  in  one.  Here  again,  little  can  be 
said  about  a  consistent  relationship  between 
productivity,  training,  or  frequency. 

-  Lew  training.  There  were  no  work  centers  for 
WUC  23ZOO  with  low  training. 

The  results  from  the  23ZOO  WUC  analyses  are 
very  similar  to  those  from  14A00.  Within  a 
particular  level  of  training,  there  may  be  a 
consistent  pattern  for  either  training  or  fre¬ 
quency  although  these  trends  are  not  consistent 
for  all  levels  of  training. 

The  principle  patterns  that  the  comparison  shows  are  those  por¬ 
trayed  on  Exhibits  III-4  and  III-5.  There  we  see  a  definite 
relationship  between  training  and  productivity. 


3 .  Examination  of  Productivity  by  Frequency 

This  section  discusses  the  productivity  by  frequency 
technique.  It  is  divided  into  two  parts: 

•  Application,  and 

•  Results. 
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a .  Application 

The  analysis  in  this  section  will  use  the  same 
productivity  and  frequency  data  from  Appendix  C  as  discussed 
previously.  The  exhibits,  one  for  each  action  code,  include  the 
appropriate  productivity  and  frequency  figures  for  each  work 
center.  The  data  points  for  all  relevant  work  centers  were  in¬ 
putted  into  a  computerized  data  base,  and  the  charts  were  print¬ 
ed.  A  standard  automated  regression  package  was  used  on  the 
data  to  obtain  the  slope  and  the  intercept  of  the  regression 
line  through  the  plotted  points. 

The  negative  value  of  a  slope  indicates  a  positive 
relationship  between  a  change  in  productivity  for  each  unit  change 
in  frequency.  The  greater  the  negative  value  (steeper  slope),  the 
greater  is  the  impact  of  frequency  on  productivity.  Conversely, 
a  positive  slope  shows  a  negative  correlation  between  productivity 
and  frequency.  Although  the  scales  for  the  various  charts  are 
different,  the  equation  of  the  line  is  given  so  the  numeric  val¬ 
ue  of  the  change  can  be  seen.  Thus,  the  sensitivity  of  frequen¬ 
cy  on  productivity  between  action  codes  can  be  measured. 

b .  Results 

This  section  will  discuss  the  results  of  the  analyses 
for  WUCs  14A00  and  23ZOO. 

(1)  Action  Codes  (14A00)  (Exhibits  III-ll  to 
111-13  ) 

Shown  below  is  a  matrix  of  values  for  the 
eight  action  codes  and  the  values  of  a  and  b  in  the  equation  for 
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the  line  fitted  through  the 

various  data 

points:  y  = 

a  +  bx. 

where  a  is  the  value  of  the 

y  intercept 

and  b  is  the 

slope  of 

the  line. 

Action 

Title 

Code 

a 

b 

Removed 

P 

57.9 

-9.2 

Removed  and  Replaced 

R 

99.5 

-3.1 

Removed  for  Cannibalization 

T 

55.5 

+6.0 

Replaced  after  Cannibalization  U 

63.4 

+2.3 

Test- Inspection-Service 

X 

74.7 

-3.1 

Troubleshoot 

Y 

187.7 

-26.3 

The  slope  of  the  regression  line  measures  the 
sensitivity,  or  change,  in  productivity  with  a  one  unit  change 
in  frequency.  For  these  six  action  codes,  the  slope  varies 
from  +6.0  (action  code  T)  to  -26.3  (action  code  Y).  In  addition 
to  T,  action  code  U  also  has  a  positive  slope — which  indicates  a 
negative  correlation  between  frequency  and  productivity  for  these 
two  action  codes.  Therefore,  of  the  six  action  codes,  frequency 
has  the  greatest  impact  on  productivity  for  action  code  Y, 
"Troubleshoot. " 


( 2 )  Action  Codes  (23Z0Q)  (Exhibit  III-14  to 
III-15  ) 

The  values  of  "a"  and  "b"  for  the  equation 

y  =  a  +  bx  are: 

Action 

Title  Code  a  b 


Removed  P 
Replaced  Q 
Test-Inspection-Service  X 


187.0 

184.9 

235.2 


+8.3 

+9.6 

-16.5 
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Exhibit  III-15.  ACTION  CODE:  X  (23ZOO) 
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The  slopes  of  the  lines  for  action  codes  P  and  Q 
are  positive  with  values  of  +8.3  and  +9.6  respectively.  This  indi¬ 
cates  a  negative  correlation  between  frequency  and  productivity  and 
also  indicates  the  effect  of  frequency  on  productivity  is  opposite 
of  what  was  expected.  Action  code  X,  "Test-Inspection-Service, " 
does  have  a  negative  slope  --  value  of  -16.5  —  as  would  be  ex¬ 
pected  . 

( 3 )  Comment 

Interestingly,  the  two  action  codes  where 
frequency  should  have  a  strong  effect,  "Y"  (14A00)  and  “X" 

(23Z00),  is  where  we  see  strong  relationships.  This  is  intui¬ 
tive  and  is  proven  by  our  data. 

4.  Use  of  ANQVA  Technique 

This  section  discusses  the  use  of  the  ANOVA  technique. 

It  is  divided  into  two  parts: 

•  Application,  and 

•  Results. 

a.  Application 

In  our  present  analysis,  we  have  observed  that 
different  frequency  levels  and  different  levels  of  training 
lead  to  different  productivity  times.  In  particular,  we  have 
observed  that  the  trend,  as  both  frequency  and  training  increase, 
is  for  productivity  to  increase .  Although  this  trend  is  observed 
for  the  data  that  is  included  in  this  analysis,  we  are  more  in¬ 
terested  in  whether  or  not  the  result  can  be  generalized  (e.g., 
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whether  or  not  this  trend  is  one  that  we  would  expect  to  see 

throughout  the  entire  Air  Force  maintenance  "population").  In 

order  to  make  this  kind  of  generalization,  a  statistical  test  is 

needed  that  will  answer  the  following  question: 

"Is  there  a  statistically  significant  difference  in  the 
observed  mean  productivity  times,  or  are  the  observed 
differences  simply  due  to  chance?" 

The  technique  that  will  answer  this  question  for 
the  variables  in  this  study  is  the  analysis  of  variance  (ANOVA) 
technique.  ANOVA  serves  to  test  the  statistical  significance  of 
sample  level  means,  or  (equivalently)  to  test  the  null  hypothesis 
that  the  sample  means  are  equal.  In  our  analysis,  this  test  is 
performed  for  both  the  frequency  and  training  variables  to  deter¬ 
mine  their  effect  on  productivity.  A  complete  description  of  the 
ANOVA  techniques  and  its  applicability  to  our  analysis  is  pro¬ 
vided  at  Appendix  C. 

Observations  from  all  action  codes  of  our  WUC 
14A00  and  WUC  23Z00  examinations  were  first  classified  into  an 
ANOVA  matrix.  The  row,  column,  and  overall  totals  and  averages 
were  then  computed  for  each  ANOVA  matrix.  These  matrices  are 
shown  as  Exhibits  III-16  and  III-17. 

Although  the  numbers  that  are  the  basis  for  these 
matrices  are  the  same  as  the  numbers  that  have  been  used  through¬ 
out  this  analysis,  they  are  combined,  and  thus  used,  in  a  very 
different  fashion. 

•  Although  each  specific  action  code  represents  a  very 
different  maintenance  job,  even  within  the  same  VvUC, 
the  ANOVA  matrices  shown  at  Exhibits  III-16  and  III-17 
include  all  of  the  action  codes  included  in  this  analy¬ 
sis  for  both  WUCs .  This  was  done  for  two  reasons.  Firs 


ANOVA  MATRIX  FOR  14A00 


Exhibit  III-17.  ANOVA  MATRIX  FOR  23Z00 


no  single  action  code  included  enough  work  center  ob¬ 
servations  to  adequately  represent  a  sample  population 
to  which  an  ANOVA  analysis  could  be  applied.  Secondly, 
the  rationale  for  accomplishing  this  kind  of  statistical 
analysis  is  to  make  some  kind  of  inference  about  the 
general  relationship  between  both  training  and  frequen¬ 
cy  to  productivity.  The  inclusion  of  all  action  codes 
makes  this  a  more  general  analysis. 

•  Averages  computed  for  each  "cell"  in  the  matrices 

shown  at  Exhibits  III-16  and  III-17  can  be  combined 
to  illustrate  what  are  known  as  the  "specific  effects" 
of  both  training  and  frequency.  These  effects  il¬ 
lustrate  the  change  in  the  average  productivity  value 
as  either  training  or  frequency  change.  Because 
cell  averages  include  all  action  codes  for  both  WUCs 
in  our  analysis,  the  specific  effects  of  training 
and  frequency  for  WUCs  provide  an  indication  of  pro¬ 
ductivity  trends  or  either  frequency  or  training  change 
by  themselves. 

Before  attempting  rigorous  statistical  analyses 
of  the  numbers  computed  in  the  ANOVA  matrices  for  the  WUCs  under 
examination,  we  translated  the  data  into  a  graphical  format  to 
allow  for  intuitive  analysis.  Exhibits  III-18  through  III-21 
show  the  general  effects  of  training  and  frequency  on  productivity 
for  both  14A00  and  23ZOO  WUCs .  All  of  these  plots  indicate  (with 
the  exception  of  a  few  points  which  contradict  the  trend)  that 
the  trend,  for  both  WUCs,  is  toward  increased  productivity  as 
either  frequency  or  training  increase.  This  is  consistent  with 
the  results  illustrated  above  (for  specific  action  codes  within 
each  WUC),  but  is  more  illustrative  because  it  includes  all  action 
codes  for  each  WUC . 

In  addition,  the  specific  effects  of  both  training 
and  frequency  can  be  gleaned  from  the  ANOVA  matrices  without  any 
further  computations.  In  fact,  the  specific  effects  are  simply 
the  averages  included  in  the  training  (row)  average  and  frequency 
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Exhibit  III-18.  ANALYSIS  OF  VARIANCE:  TRAINING  EFFECTS  (14AOO) 
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Exhibit  III-21 .  ANALYSIS  OF  VARIANCE:  FREQUENCY  EFFECTS  (23ZOO) 


(column)  average  cells  in  the  ANOVA  matrices.  Examining  these 
numbers  indicates  that  the  specific  effect  of  training  for  both 
the  14A00  and  23Z00  WUC  is  positive  (i.e.,  as  training  increases, 
productivity  increases).  The  specific  frequency  effect  for  23Z00 
illustrates  the  same  result.  The  specific  frequency  effect  in 
the  case  of  14A00,  however,  is  ambiguous.  As  frequency  goes 
from  the  "low"  range  (_<  1  action/worker)  to  the  "medium"  range 
(1  <  actions/worker  <_  2) ,  productivity  decreases.  As  frequency 
increases  to  the  high  range  (2  <  actions/worker),  productivity 
increases.  Therefore,  the  productivity  trend  in  this  case  is 
not  clearcut.  This  result  could  lead  one  to  conclude  that  fre¬ 
quency,  at  least  as  we  have  defined  it,  has  little  impact  on 
productivity,  or  that  one  or  more  erroneous  observations  in  the 
medium  frequency  range  have  thrown  off  our  results. 

b.  Results 

The  observations  described  in  the  previous 
section  with  regard  to  our  ANOVA  technique  of  analysis,  or  our 
other  analyses  described  previously,  pertain  only  to  the  sample 
of  Air  Force  maintenance  work  centers  we  examined.  We  cannot 
correctly  state  that  the  trends  observed  are  indicative  of  the 
entire  Air  Force  maintenance  population  without  statistical  just¬ 
ification.  This  justification  is  either  proved  or  disproved 
by  the  complete  application  of  the  ANOVA  technique  (as  des¬ 
cribed  in  Appendix  C). 


Exhibits  III-22  and  III-23  summarize  the  statistics 
associated  with  the  ANOVA  matrices  for  14A00  and  23Z00.  The 
pertinent  comparative  statistics  in  these  two  tables  are  the  F- 
ratios  for  training  and  frequency.  Comparing  these  ratios 
with  a  standard  F-statistic,  derived  from  the  standard  F-dis- 
tribution  at  a  specific  level  of  confidence,  allows  us  to  draw 
conclusions  as  to  the  statistical  validity  of  our  analytical 
results.  In  particular,  if  the  F-ratios  shown  in  Exhibits  III-22 
and  III-23  are  larger  than  the  appropriate  F-statistics  from 
the  standard  F-distribution,  then  we  can  conclude  that  the  re¬ 
lationships  that  we  have  observed  in  this  analysis  are  valid 
for  Air  Force  maintenance  in  general.  Training  does  lead 
to  a  general  increase  in  productivity  and  frequency  leads  to  a 
general  increase  in  productivity. 

The  appropriate  tests  for  both  14A00  and  23Z00 
training  and  frequency  variables  are  explicitly  provided  below: 
14A00 

•  Training 

F  (training)  =  2.9 

F  statistic  (from  F  distribution,  90%  confidence, 

2  d.f.  for  training  variance,  4  d-f.  for  error 
variance)  =  4.32 

-  2.9  <4.32,  so  the  relationship  between  training 

and  productivity  observed  in  our  14A00  analysis 
does  not  indicate,  at  the  90%  confidence  level, 
that  training  has  a  positive  impact  on  productivity. 

•  Frequency 


F  (frequency)  =  0.7 
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Exhibit  III-23 .  ANOVA  TABLE  FOR  23Z 


F  statistic  (from  F  distribution,  90%  confidence, 

2  d.f.,  for  frequency  variance,  4  d.f.  for  error 
variance)  =4.32 

0.7  <  4.32,  so  the  relationship  between  frequency 
and  productivity  observed  in  our  14A00  analysis 
does  not  indicate,  at  the  90%  confidence  level, 
that  frequency  has  a  positive  impact  on  productivity. 


23Z00 

•  Training 

-  F  (training)  =  19.1 

-  F  statistic  (from  F  distribution,  90%  confidence, 

1  d.f.  for  training  variance,  2  d.f.  for  error 
variance)  =  8.53 

19.1  >  8.53,  so  the  relationship  between  training 
and  productivity  observed  in  our  23Z00  analysis  does 
indicate,  at  the  90%  confidence  level,  that  training 
has  a  positive  impact  on  productivity. 

•  Frequency 

F  (frequency)  =  1.4 

F  statistic  (from  F  distribution,  90%  confidence, 

2  d.f.  for  frequency  variance,  2  d.f.  for  error 
variance)  =  9.0 

-•  1.4  <  9.0,  so  the  relationship  between  frequency 

and  productivity  observed  in  our  23Z00  analysis 
does  not  indicate,  at  the  90%  confidence  level, 
that  frequency  has  a  positive  impact  on  productivity 

As  described  above,  the  statistical  results  of  our 
analysis,  provided  by  the  application  of  the  ANOVA  technique,  show 
in  three  out  of  four  tests  that  our  results  are  not  indicative  of 
the  overall  Air  Force  maintenance  population.  Several  obser¬ 
vations  should  be  noted,  however,  that  are  not  directly  apparent 
from  these  statistical  results. 

•  Training,  in  both  WUC  examinations,  has  a  much  larger 
impact  on  productivity  than  frequency.  The  F-ratio 
is  a  measure  of  "how  much  change"  in  the  dependent 
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variable  (productivity)  is  attributed  to  an  indepen¬ 
dent  variable.  The  larger  the  F-ratio  is,  the  more 
change  can  be  attributed  to  the  independent  variable 
in  question.  In  both  the  14A00  and  the  23Z00  examin¬ 
ations,  the  training  F-ratio  is  much  larger  than  the 
frequency  F-ratio. 

•  The  number  of  work  center  observations  within  each 

"cell"  in  the  two  ANOVA  matrices  were  not  equal;  there¬ 
fore,  the  cell  averages  shown  in  these  two  matrices 
are  based  on  different  numbers  of  observations.  This 
statement  is  not,  in  itself,  a  qualifier  of  the  results 
that  we  obtained;  however,  some  statistics  texts  allude 
to  the  fact  that  this  case  could  lead  to  a  bias  in  the 
results  that  are  calculated.  The  application  of  this 
technique,  taking  unequal  observations  into  consider¬ 
ation,  would  require  computer  services  that  are  un¬ 
available  to  us  at  the  present  time.  In  future  examin¬ 
ations,  this  capability  may  be  available  and  more  ac¬ 
curate  results  could  show  that  both  factors  are  statis¬ 
tically  significant. 


5 .  Observations 

Our  analysis  used  three  techniques  to  examine  productivity. 
Each  of  them  produced  some  positive  analytical  results.  The 
techniques  used  were:  productivity  by  action  code,  productivity  by 
frequency,  and  ANOVA. 

The  productivity  by  action  code  technique  allowed  us 
to  examine  the  effect  training  had  on  productivity  in  a  graphic 
form.  The  results  appeared  to  show  that  for  both  of  the  WUCs 
examined  (14A00  and  23ZOO)  the  effect  of  training  is  significant 
in  terms  of  productivity  increases.  This  was  obvious  in  our 
"wing-to-wing"  comparisons.  The  attempt  to  group  work  centers 
by  training  status  (high,  medium,  low),  and  thus  infer  some 
meaning  concerning  the  effect  of  frequency,  did  not  provide 
useful  results.  Any  relationship,  holding  training  relatively 
constant,  between  frequency  and  productivity  was  not  obvious. 
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The  productivity  by  frequency  technique  plotted 
frequency  versus  productivity.  A  regression  line  was  fitted  to 
each  plot  and  the  results  for  WUC  14A00  showed,  in  four  out  of 
six  cases  (six  action  codes),  a  positive  correlation  (negative 
slope)  between  frequency  and  productivity.  The  results  for 
WUC  23Z00  were  not  clear.  We  examined  three  action  codes  and 
in  two  cases  got  a  negative  correlation  between  frequency  and 
productivity.  In  one  case  we  got  a  positive  correlation. 

Overall,  the  technique  appears  to  show  a  positive  relationship 
between  frequency  and  productivity. 

The  ANOVA  technique  allowed  us  to  examine  the  impact 
of  both  frequency  and  training  on  maintenance  productivity.  The 
statistical  results  were  mixed  since  in  three  out  of  four  tests 
it  was  not  indicated  that  these  results  were  indicative  of  the 
overall  Air  Force  maintenance  population.  However,  it  must  be 
noted  that  training,  in  both  WUC  examinations,  has  a  much  larger 
effect  on  productivity  than  frequency.  In  the  case  of  WUC  23Z00, 
there  was  a  positive  indication  of  a  relationship  between  train¬ 
ing  and  productivity  at  the  90%  confidence  level. 

In  order  to  assure  ourselves  that  the  amount  of  time 
spent  by  work  centers  on  the  actions  we  examined  was  representa¬ 
tive,  we  did  a  limited  comparison  of  actions  we  examined  to  total 
actions  worked  on.  There  are  28  system  level  WUCs  for  the  F-16. 
WUC  14000  was  the  highest  manhour  consumer  in  our  sample  (10.6%), 
WUC  23000  was  fourth  with  8.0%.  There  are  seven  subsystems  within 
WUC  14000 — WUC  14A00  was  32%  of  WUC  14000  or  3.4%  of  total  wing 
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manhours.  There  are  twelve  subsystems  within  WUC  23000 — WUC  23Z00 
was  24%  of  WUC  23000  or  2.0%  of  total  wing  manhours.  Thus  out  of 
113  subsystem  WUCs  the  two  WUCs  we  examined  (14A00  and  23Z00)  are 
quite  representative  of  total  wing  maintenance  since  they  consume 
over  5%  of  total  maintenance  manhours  in  the  sample  we  looked  at. 

Our  intention  was  to  show  a  relationship  between  maintenance 
productivity  and  installation-level  training.  We  chose  courses 
taught  using  simulators  for  our  examination  of  training,  but  did 
not  compare  simulator  training  with  non-simulator  training. 
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IV.  AN  EXAMINATION  OF  THE  IMPACT  OF  INSTALLATION-LEVEL  TRAINING 
ON  ARMY  OPERATIONAL  UNIT  MAINTENANCE  PRODUCTIVITY 

This  section  discusses  the  impact  of  installation-level 
training  on  Army  operational  unit  productivity.  The  analysis 
will  develop  a  linkage  between  individual  training  conducted 
at  the  installation  level  and  unit  productivity. 

Unlike  the  Air  Force,  the  Army  does  not  conduct  formalized 
installation  training.  However,  under  the  auspices  of  the  U.S. 
Army  Training  and  Doctrine  Command  (TRADOC),  school-developed 
training  materials  are  furnished  to  operational  units  to  conduct 
their  own  training.  TRADOC,  through  its  Training  Effectiveness 
Analysis  (TEA)  process,  has  a  means  of  evaluating  training.  The 
TEA  process  is  conducted  by  the  TEA  division  of  the  TRADOC  Sys¬ 
tems  Analysis  Activity  (TRASANA)  located  at  White  Sands  Missile 
Range,  New  Mexico.  TRASANA  conducts  specialized  training  effec¬ 
tiveness  analyses,  normally  in  conjunction  with  the  fielding  of 
new  major  weapons  systems  such  as  the  Ml  (ABRAMS)  tank. 

Individuals  who  are  required  to  operate  and  maintain  weapons 
systems  reach  the  journeyman  level  through  individual  training 
conducted  at  the  installation  level.  Service  schools,  by  design, 
may  not  teach  certain  tasks,  which  then  become  a  unit  level 
responsibility.  Many  tasks  are  best  taught  at  the  installation 
level  because  job  tasks  vary  greatly  due  to  factors  peculiar  to 
a  unit,  such  as  models  of  equipment,  and  particular  maintenance 
problems  which  are  location-oriented.  Therefore,  installation 
schools  are  a  key  component  in  the  individual  training  process. 
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An  installation  school  can: 

•  assist  the  unit  commander  in  solving  his  training 
problems  ; 

•  provide  a  valuable  source  of  information  to  the 
command  conducting  the  training;  and 

•  serve  as  a  vehicle  to  distribute  the  desired  level  of 
support  for  individual  training  at  the  unit. 

The  Army  has  locally-developed  and  -supported  installation 
schools  at  most  major  troop  locations.  This  training  is  funded 
out  of  operational  resources  rather  than  training  resources.  Be¬ 
cause  installation  schools  often  take  instructors  from  operation¬ 
al  units,  organizations  are  reluctant  to  divert  personnel  from 
operational  units  to  serve  as  instructors.  Therefore,  an  instal¬ 
lation  school  may  not  receive  a  high  priority  for  its  manning 
requirements.  One  solution  that  the  Army  has  used  to  alleviate 
this  problem  is  the  use  of  contract  training.  Contractor-pro¬ 
vided  training,  supplementing  formal  training  and  used  to  spe¬ 
cifically  address  installation/unit  needs,  is  utilized  in  two 
different  variants. 

•  Contractors  provide  training  according  to  an  installa¬ 
tion-developed  course  of  instruction,  thus  acting  as 
civilian  replacements  for  military  instructors.  Many 
local  colleges  or  universities  in  the  vicinity  of 
military  installations  can  provide  this  kind  of  service 
For  example,  North  Texas  State  University  is  under 
contract  to  provide  instructors  for  certain  courses 
(logistics;  organizational  maintenance)  to  Fort  Carson, 
Colorado. 

•  Contractors  provide  training  on  equipment  that  their 
company  manufactures.  Training  is  geared  to  the  needs 
of  the  units  at  the  installation.  To  ensure  that  the 
latest  equipment  information  is  incorporated,  the  ac¬ 
tual  curriculum  is  developed  by  the  contractor  himself. 
An  example  of  this  type  of  training  is  the  courses 


provided  by  Stewart  Stevenson  Power,  Inc.  for  main¬ 
tenance  mechanics.  Stewart  Stevenson  is  a  subsidiary 
of  Detroit  Diesel  Allison,  Inc.,  which  manufactures 
the  engines  that  are  in  a  large  number  of  the  vehicles 
on  which  mechanics  work. 

An  examination  was  made  of  operational  units  in  which  unit 
schools  were  present  and  for  which  accessible  information  con¬ 
cerning  maintenance  productivity  was  available.  This  latter  in¬ 
formation  need  required  a  means  of  measuring  individual  or  team 
productivity  within  the  unit.  Therefore,  a  data  source  was  re¬ 
quired  that  provided  this  capability.  Once  this  source  was  iden¬ 
tified,  the  available  data  was  linked  with  installation-level 
training  data,  and  a  unit  level  training/productivity  analysis 
was  developed. 

This  section  contains  three  subsections: 

•  Skill  Training  Evaluation  Approach, 

•  Skill  Training  Evaluation  Results,  and 

•  Overview  of  the  Army  Training  Effectiveness  Analysis 
Process . 

Sections  IV. A  and  IV. B,  Skill  Training  Evaluation  Approach  and 
Skill  Training  Evaluation  Results,  discuss  the  examination  of 
installation-level  training  at  two  different  installations  and 
the  relationship  of  this  training  to  unit  productivity.  Section 
IV. C,  Overview  of  the  Army  Training  Effectiveness  Analysis  Process, 
discusses  the  role  TRASANA  plays  in  examining  Army  training  in 
the  field  and  the  general  procedures  they  use  in  conducting 
evaluations . 

Supporting  data  is  presented  in  Appendix  D,  Training  Effec¬ 
tiveness  Analysis  Background  Materials. 
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SKILL  TRAILING  EVALUATION  APPROACH 
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The  focus  of  this  study  is  the  impact  of  training  conducted 
at  the  installation  level  (either  school  training  or  contractor 


training)  on  maintenance  job  performance.  The  typical  training 
path  for  Army  enlisted  maintenance  personnel  is  shown  on  Exhibit 
IV— 1 .  Enlistees  attend  basic  training  and  advanced  individual 
training  (AIT)  prior  to  assignment.  During  AIT,  soldiers  learn 
the  basic  skills  required  within  their  Military  Occupational 
Specialty  (MOS).  After  AIT,  soldiers  are  assigned  to  a  unit, 
where  training  is  continued  by  OJT.  OJT  at  the  unit  level  is 
supplemented  as  needed  by  installation-level  school  training 
and/or  contractor  training. 


This  section  discusses  the  following  topics: 

•  individual  training  conducted  by  installation-level 
schools  and  contractor  courses,  and  the  relationship 
of  this  training  to  unit  maintenance  training  needs; 

•  data  sources  which  are  available  to  provide  maintenance 
performance  data; 

•  unit  selection;  and 

•  MCR's  Army  skill  training  evaluation  methodology. 


1 .  Installation  Level  Individual  Training 

The  primary  functions  of  installation-level  training 
include  the  following: 

•  to  supplement  the  generalized  training  received  in 
formal  schools  with  equipment-specific  training 
related  to  types  of  equipment  located  at  that 
installation ; 

•  to  provide  transition  training  to  personnel  whose 
previous  experience  has  been  on  other  models  of 
equipment;  and 
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Exhibit  IV-1.  TYPICAL  TRAINING  PATH  OF 
ARMY  ENLISTED  MAINTENANCE  PERSONNEL 


•  to  provide  upgrade  and  refresher  training. 

In  the  Army,  installation-level  training  is  conducted 
by  an  installation-level  school  or  by  a  contractor.  Army  in¬ 
stallation-level  schools  are  established  by  the  installation  using 
its  own  resources. 

Contractor  courses  are  also  funded  by  the  installation, 
using  its  own  resources.  Contract  courses  either  augment  or 
replace  installation-level  school  courses  where  they  are  used. 

The  research  documented  by  this  technical  report  focuses 
on  both  installation-level  school  training  and  contractor  training 
which  support  Army  direct  support  (DS)  maintenance  battalions 
in  two  different  mechanized  infantry  divisions.  Direct  support 
maintenance  is  the  intermediate  level  in  the  Army's  maintenance 
hierarchy.  The  three  levels  of  maintenance  for  non-aircraft 
systems  are  discussed  below. 

•  Organizational  maintenance:  this  is  the  most  direct 
level  of  maintenance  in  the  Army.  At  this  level, 
the  operational  unit  itself  performs  maintenance 
actions  such  as  tune-ups,  oil  changes,  and  other 
simple  maintenance  tasks. 

•  Direct  support  ( DS )  maintenance:  this  category 
consists  of  maintenance  tasks  of  medium  difficulty, 
such  as  replacing  engine  components,  transmissions,  or 
other  such  maintenance  tasks.  DS  maintenance  is 
usually  performed  by  a  separate  maintenance  battalion 
within  a  division. 

•  General  support  (GS)  maintenance:  this  is  the  highest 
level  of  maintenance  performed  at  the  installation 
level  in  the  Army.  GS  tasks  include  major  engine 
overhaul,  engine  rebuilding,  and  other  related  main¬ 
tenance  tasks.  GS  maintenance  is  performed  at  an 
installation  level  GS  shop,  which  receives  GS  jobs 
from  all  units  that  are  assigned  to  the  installation. 


Within  a  specific  DS  unit,  work  is  performed  in  several  different 
companies  which  include  soldiers  with  a  variety  of  different 
MOSs.  Team  maintenance  (two  or  more  individuals  performing  a 
single  maintenance  task)  is  standard  procedure  in  most  DS  units. 
Cross-skill  maintenance  (personnel  who  are  trained  in  one  skill 
area  performing  a  maintenance  task  associated  with  a  different 
skill  area)  is  performed  as  required  by  workload  and  personnel 
shortages,  particularly  within  the  63-series  MOSs  (vehicle 
mechanics ) . 

2 .  Data  Sources 

Three  different  sources  of  maintenance  performance  data 
were  examined  in  the  present  analysis: 

•  The  Army  Maintenance  Management  System  (TAMMS); 

•  the  Maintenance  Performance  System  (MPS);  and 

•  unit  level  records  (DA  Form  2407). 

a.  TAMMS 

The  Army  Maintenance  Management  System  is  an 
equipment  records  system  used  for  controlling  the  operation  and 
maintenance  of  Army  materiel.  TAMMS  captures  certain  maintenance 
information  recorded  on  Army  maintenance  request  forms,  and  the 
system  produces  a  variety  of  different  maintenance  reports  and 
summaries  from  this  information.  The  general  purpose  of  TAMMS 
is  to  provide  the  information  needed  to  manage  the  maintenance 
of  weapons  and  support  equipment,  the  availability  of  spare  parts, 
and  other  related  functions. 
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b . 

The  Maintenance  Performance  System  is  a  training 
needs  information  system  that  has  been  developed  by  the  Army 
Research  Institute  (ARI).  The  MPS  is  currently  under  test  as  a 
part  of  a  research  program  in  an  Army-wide  effort  to  improve 
maintenance.  The  stated  purpose  of  the  MPS  is  to  "help  identify 
problems  related  to  lack  of  technical  skills,  poor  utilization 
of  training  resources,  and  poor  shop  management . Mainten¬ 
ance  performance  data  is  integral  to  the  accomplishment  of  this 
stated  purpose. 


c .  Unit  Level  Records 

DS  maintenance  units  maintain  records  in  each 
company  shop  office  on  maintenance  actions  performed.  These 
records  consist  of  copies  of  DA  Form  2407,  "Maintenance  Request, 
that  an  operational  unit  is  required  to  submit  for  maintenance 
work  to  be  performed.  In  addition  to  including  information  on 
the  equipment  submitted  for  maintenance  and  the  actual  mainten¬ 
ance  work  performed  on  the  equipment,  the  time  for  completing 
each  maintenance  action  is  also  noted  on  this  form.  The  infor¬ 
mation  on  this  form  is  used  in  TAMMS. 

3 .  Unit  Selection 

Unit  selection  was  limited  in  the  present  study  due  to 
the  fact  that  one  of  our  data  sources,  the  MPS,  is  currently  a 
test  system  and  is  only  available  at  two  different  locations. 

5/Maintenance  Performance  System;  User's  Reference  Manual, 

U.s.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences,  January  1981. 
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These  locations  are: 

•  Fort  Carson,  Colorado,  and 

•  Fort  Polk,  Louisiana. 

The  identity  of  the  installation  is  not  relevant; 
hence,  we  have  re-named  them  Fort  R  and  Fort  S.  The  two  main¬ 
tenance  battalions  are  numbered  R123  and  S789,  respectively. 

This  alpha-numeric  code  is  used  throughout  the  analysis  portion 
of  this  section  to  identify  the  units. 

4 .  Methodology 

The  methodology  that  was  followed  in  performing  our 
Army  skill  training  evaluation  is  shown  in  Exhibit  IV-2  and  ex¬ 
plained  in  more  detail  below. 

Step  1:  Determine  units  to  be  examined  in  analysis. 
Two  installations  were  chosen  as  data  sources  for  this 
study.  DS  maintenance  units  were  chosen  as  the  focus  for  this 
analysis  for  two  reasons: 

•  DS  maintenance  personnel  typically  perform  maintenance 
actions  that  could  be  taught  to  them  through  installa¬ 
tion  level  training,  and 

•  the  MPS  focused  on  DS  units  at  the  two  installations 
chosen  for  data  collection. 

The  DS  maintenance  units  at  the  two  installations  chosen  are: 

•  the  704th  Maintenance  Battalion  at  Fort  Carson,  and 


the  705th  Maintenance  Battalion  at  Fort  Polk. 


Determine  Units  To 
Be  Examined  in  Analysis 


Exhibit  IV- 2.  ARMY  SKILL  TRAINING 
EVALUATION  METHODOLOGY 
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Step  2 : 


Request  installation  school/contractor  course 
information  for  individuals  in  the  units 
selected  for  analysis;  determine  individual 
training  status. 

Once  the  units  to  be  analyzed  were  identified,  course 
information  was  required  to  determine  the  installation-level 
training  status  of  maintenance  individuals  in  each  unit.  DS  main¬ 
tenance  battalions  in  a  mechanized  infantry  division  include  a 
large  number  of  vehicle  mechanics,  who  do  most  of  the  main¬ 
tenance  for  the  unit.  Course  information  requests  were  restricted 
to  those  courses  which  dealt  with  vehicle  mechanics  (63-series 
MOSs ) .  This  restriction  was  made  to  limit  the  amount  of  data 
that  had  to  be  requested. 

Once  course  data  was  obtained,  individuals  in  the  63- 
series  MOSs  in  each  unit  were  assigned  a  training  status  of  either 
trained  or  untrained.  For  the  purposes  of  this  analysis,  these 
determinations  were  defined  as  follows: 

•  trained  —  an  individual  must  have  completed  an 
installation  level  course  pertaining  to  a  63-series 
MOS  ?  and 

•  untrained  --  all  other  individuals  who  have  not  com¬ 
pleted  a  63-series  MOS  course  at  the  installation 
level . 

Step  3:  Associate  productivity  data  with  training 
information . 

At  this  point,  productivity  data  was  to  be  linked  with 
individual  training  status.  Trained  individuals  and  untrained 
individuals  could  be  segregated,  and  productivity  information 
for  specific  maintenance  actions  could  be  compared  between  the 
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two  groups  using  statistical  techniques.  As  will  be  discussed 
in  the  next  section,  we  were  unable  to  accomplish  this  step  due 
to  a  lack  of  appropriate  data. 

B.  SKILL  TRAINING  EVALUATION  RESULTS 

This  section  discusses  the  following  topics: 

•  application  of  the  skill  training  evaluation  method¬ 
ology  and  limitations  which  affected  the  application 
of  the  methodology;  and 

•  observations  on  improvements  that  can  be  made  to  allow 
for  improved  training/productivity  linkage  evaluations. 

1 .  Application 

The  MCR  project  team  gathered  productivity  data  from 
Fort  Carson  and  Fort  Polk.  Course  data  was  requested  from  both 
installation  level  schools  that  pertained  to  63-series  MOSs ,  and 
on  the  Detroit  Diesel  Allison  courses  that  are  taught  to  indi¬ 
viduals  at  Fort  R  (Fort  S  does  not  supplement  installation-level 
school  courses  with  contractor  courses).  We  then  attempted  to 
process  this  data  as  described  in  our  skill  training  evaluation 
methodology  (Section  IV. A. 4).  Problems  with  this  application 
became  apparent  and  so  a  full  application  of  our  methodology  was 
not  possible. 

An  analysis  such  as  this  is  only  as  good  as  the  data  on 
which  it  is  based.  Unfortunately,  none  of  the  data  sources  that 
were  identified  as  potentially  useful  for  this  study  proved  to 
be  appropriate.  Different  shortcomings  limited  the  use  of  each 
of  the  data  sources  identified. 
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two  groups  using  statistical  techniques.  As  will  ue  discus 
in  the  next  section,  we  were  unable  to  accomplish  this  step 
to  a  lack  of  appropriate  data. 

B  .  SKILL  TRAILING  h VALUATION  RLSULTS 

This  section  discusses  the  following  topics: 

•  application  of  the  skill  training  evaluation  met:, 
ology  and  limitations  which  affected  the  appiicat 
of  the  methodology ;  and 

•  observations  on  improvements  that  can  be  made  to 
for  improved  training/productivity  linkage  evtiua 

1 .  Appl 1 cat ion 

The  MCR  project  team  gathered  productivity  cate  r 
tort  Carson  and  Fort  Polk.  Course  data  was  requested  from 
installation  level  schools  that  pertained  to  03-series  MObs 
on  the  Detroit  Diesel  Allison  courses  that  are  taught  to  in 
viduals  at  Fort  K  (Fort  S  does  not  supplement  installation- 
school  courses  with  contractor  courses).  We  then  attempted 
process  this  uaca  as  described  in  our  skill  training  ovaluu 
methodology  (Section  IV. A. 4).  Problems  with  this  applicate 
became  apparent  and  so  a  full  application  of  our  met hod clog 
no  .  possible. 

An  analysis  such  as  this  is  only  as  good  as  che  : 
which  it  is  based.  Unfortunately,  none  of  the  data  sources 
were  identified  as  potentially  useful  for  this  study  proved 
ne  appropriate.  Different  shortcomings  limited  the  use  >! 
of  the  data  sources  identified. 
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a. 


TAMMS  Data  Limitations 


A  previous  study  has  documented  the  inadequacies 
of  TAMMS  data  for  training  effectiveness  purposes The  TAMMS 
system  was  designed  to  provide  information  for  purposes  of  main¬ 
tenance  and  logistics;  not  for  the  purpose  of  evaluating  mainten¬ 
ance  performance  by  individuals.  The  biggest  problem  with  this 
database  for  our  purposes  is  that  the  names  of  individuals  who 
perform  maintenance  actions  are  not  maintained  in  the  central  data 
files.  Because  the  ability  to  identify  and  track  individuals 
is  central  to  our  methodology  for  relating  maintenance  perfor¬ 
mance  to  training,  TAMMS  data  is  not  appropriate  for  our  analysis. 

b.  MPS  Data  Limitations 

In  theory,  the  MPS  database  is  an  ideal  source  of 
data  for  any  training/productivity  analyses.  MPS  data  includes 
information  which  directly  identifies  individuals  that  are  per¬ 
forming  specific  maintenance  tasks.  Therefore  individuals  can  be 
readily  identified,  training  information  can  be  associated  with 
names,  and  performance  can  be  tracked. 

The  MPS  system  is  still  being  tested.  however,  at 
the  two  installations  at  which  it  is  in  operation,  the  current 
databases  are  very  small.  In  addition,  the  DS  level  MPS  test 
phase  has  been  completed  at  Fort  R  and  DS  data  was  no  longer 
input  after  1  October  1982.  These  factors  proved  to  be  a  problem 


6/hvaluatinq  the  Effectiveness  of  Maintenance  Training  by  Using 
Currently  Available  Maintenance  Data,  IDA,  August  1981. 


when  a  training  data/productivity  data  linkage  was  attempted. 

While  trying  to  link  training  information  with  associated  pro¬ 
ductivity  data  in  the  MPS  databases,  we  discovered  that  most  of 
the  names  on  the  training  list  were  not  included  in  the  MPS 
databases.  Specifically,  we  found  the  following. 

•  At  Fort  R,  45  names  were  received  as  63-series  MOSs 
from  the  SI 23d  Maintenance  Battalion  who  had  been 
trained  through  the  installation  school  or  through  the 
Detroit  Diesel  Allison  contractor  course  in  the  past 
year.  Of  this  list  of  names,  only  four  (9%)  were  found 
to  be  included  in  the  MPS  database. 

•  At  Fort  S,  10  names  were  received  as  63-series  MOSs 
from  the  S789th  Maintenance  Battalion  who  were  trained 
through  the  installation  school  in  the  past  year.  This 
sample  size  itself  was  so  small  as  to  bring  to  question 
the  validity  of  any  statistical  analyses  based  on  this 
data,  and  only  one  name  (10%)  was  found  to  be  included 
in  the  MPS  database. 

Therefore,  the  small  size  of  the  MPS  databases  at 
our  source  installations  effectively  prohibited  an  effective  train¬ 
ing/productivity  linkage. 


c .  Unit  Level  Data  Limitations 

DA  Form  2407,  "Maintenance  Request,"  records  are 
retained  by  the  DS  unit  for  all  maintenance  actions  performed  at 
this  level.  However,  certain  shortcomings  of  the  data  included 
on  these  forms  limit  its  use  in  the  present  study. 

•  "Work  performed"  information  is  very  generic.  Mechanics 
are  required  to  account  for  the  work  they  performed  on 
the  2407,  but  such  notation  is  most  commonly  made  at 
the  broadest  level  possible.  For  example,  any  work 
performed  on  an  engine  or  its  components,  ranging  from 
simple  tasks  to  the  most  complex  ones,  are  commonly 
represented  as  "engine  work"  on  the  2407.  Data  at 
this  generic  level  of  detail  is  inappropriate  for  the 
type  of  analysis  represented  by  this  study. 
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•  Data  is  not  maintained  for  time  productivity  purposes; 
it  is  maintained  to  keep  track  of  parts  used,  parts 
needed,  etc.  Therefore,  time  information  that  is 
recorded  on  the  2407  cannot  be  assumed  to  be  truly 
accurate . 

•  Individual  names  of  maintenance  personnel  are  not 
shown  on  the  form. 

In  addition  to  these  intrinsic  limitations  of  the 
2407  data  itself,  there  is  another  external  factor  that  serves  to 
limit  the  statistical  applicability  of  this  data. 

•  DS  units  maintain  2407  records  for  90  days;  data  prior 
to  90  days  is  thrown  out.  The  range  of  the  data  is 
thus  very  limited. 


2 .  Observations 

Data  limitations  discussed  above  limit  the  conclusions 
that  could  be  made  with  respect  to  the  results  of  this  analysis. 

No  specific,  quantitative  observations  or  conclusions  can  be  ad¬ 
vanced  concerning  the  relationship  between  installation  level 
maintenance  training  and  productivity. 

Subjectively,  however,  installation-level  training  does 
seem  to  have  a  positive  impact  on  maintenance  productivity.  Inter¬ 
views  were  conducted  with  several  individuals  (ranging  from  DS 
mechanics  to  staff  officers  at  the  division  level)  at  Fort  Carson. 
All  of  these  individuals  had  the  same  impression  of  installation- 
level  training:  although  the  positive  benefit  of  the  training  may 
not  be  quantifiable,  the  benefit  does  exist.  Mechanics  were  able 
to  "diagnose  problems  better"  and  "p  'form  troubleshooting  actions 
with  more  accuracy"  as  the  result  of  installation-level  training 
(in  this  case,  Detriot  Diesel  Allison  courses). 
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These  subjective  observations  are  all  that  can  be  said, 
at  this  time,  concerning  the  relationship  upon  which  this  analysis 
has  focused.  Current  databases  from  which  information  can  be 
obtained  for  training/productivity  analyses  proved  inadequate  for  a 
specific,  quantitative  analysis.  Although  the  current  databases 
are  not  appropriate  for  the  kinds  of  analyses  that  we  are  at¬ 
tempting,  this  will  not  necessarily  be  the  case  in  the  future. 

The  Army  is  developing  the  systems  to  keep  track  of 
productivity  information.  When  they  are  completed,  the  present 
type  of  analysis  could  be  successful.  In  particular,  the  fol¬ 
lowing  data  sources  could  provide  appropriate  information. 

•  The  MPS .  As  was  mentioned  above,  the  MPS  is  currently 
in  the  test  mode.  As  more  data  is  collected  by  this 
system  at  Fort  S,  and  if  the  system  is  expanded  and 
data  is  collected  at  other  Army  installations,  this 
system  could  prove  to  be  a  very  effective  training 
management  tool,  especially  for  the  present  type  of 
analysis . 

•  The  Standard  Army  Maintenance  System  (SAMS).  The  SAMS 
is  an  automated  maintenance  management  system  that  will 
relace  the  current  TAMMS  system  and  encompass  all  levels 
of  Army  materiel  maintenance.  SAMS  will  improve  upon 
the  present  TAMMS  system  in  that  a  maintenance  job  will 
be  "tracked"  on  an  in-shop  computer  as  it  progresses 
through  work  stages,  and  each  different  stage  of  work 
will  be  be  explicitly  noted  in  the  job  record.  There¬ 
fore,  the  records  should  be  more  accurate  than  those 

in  TAMMS  (which  is  automated  at  a  much  higher  level) 
and  include  more  detailed  data  on  particular  actions 
performed.  Unfortunately,  the  SAMS  system  is  also  not 
designed  for  training  analysis  purposes;  no  information 
that  identifies  individuals  is  included  in  this  system. 
The  system,  however,  is  still  in  preliminary  implement¬ 
ation  stages.  Data  elements  could  theoretically  be 
added  to  the  system  if  a  strong  rationale  were  given 
for  their  inclusion.  Even  if  individual  identification 
were  not  included  in  the  system,  the  improvements  in 
accuracy  and  level  of  detail  over  the  TAMMS  database 
could  be  of  benefit  for  training/productivity  analyses. 
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A  "macro"  level  approach,  which  specifically  identi¬ 
fies  certain  types  of  installation-level  training  with 
certain  types  of  maintenance  actions,  would  be  much 
easier  to  accomplish  if  maintenance  actions  were  identi¬ 
fied  more  explicitly  in  an  automated  database.  SAMS 
could  provide  this  capability,  whereas  the  current 
TAMMS  does  not. 

C.  OVERVIEW  OF  THE  ARMY  TRAINING  EFFECTIVENESS  ANALYSIS  PROCESS 

The  focus  of  the  MCR  study  is  the  impact  of  training 
conducted  at  the  installation  level  on  maintenance  job  per¬ 
formance.  The  information  in  this  section  concerning  TRASANA 
was  felt  to  be  analogous  to  our  efforts  and  thus  worthy  of 
explanation. 

This  section  discusses  the  overall  Army  Training  Effective¬ 
ness  Analysis  (TEA)  process.  More  detailed  information  on  the 
structure  of  the  TEA  process  and  on  an  example  TEA  for  the  Ml 
(ABRAMS)  tank  is  in  Appendix  D,  Training  Effectiveness  Analysis 
Background  Materials. 

The  mission  of  TRASANA  is  to  serve  as  an  analytical  center 
for  combat  and  training  developments  and  to  conduct  analyses. 

The  TEA  Division  of  TRASANA  is  the  single  agency  for  managing 
the  TEA  process  and,  as  such,  conducts  TEAs  and  reviews  TEA  stud¬ 
ies  conducted  by  others.  This  section  will  discuss  the  goals 
and  emphasis,  support  provided,  and  process  used  by  the  TEA 
Division. 

1 .  Goals  and  Emphasis 

Training  effectiveness  analysis  is  a  process  for  which 
TRASANA 's  TEA  Division  is  the  proponent.  The  TEA  Division  has 
as  its  goals  and  areas  of  emphasis  the  following: 
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•  contribute  to  the  fielding  of  optimum  training  pack¬ 
ages  with  the  hardware? 

•  assist  in  the  improvement  of  existing  training 
packages ; 

•  build  a  TEA  data  base;  and 

•  emphasize  Army  Force  Modernization  by: 

-  conducting  more  TEA  on  post-fielded  systems; 

-  conducting  TEA  early  in  the  developmental  phase; 
and 

emphasizing  the  permeation  of  the  TEA  process 
throughout  the  Army. 

The  process  is  carried  out  through  evaluations  in  four  major 
categories:  training,  training  devices,  soldier/hardware  inter¬ 

face,  and  evaluation  of  MOS  selection  criteria.  The  evaluations 
may  include  one  or  more  of  these  categories.  Cost  analysis  may 
also  be  included  in  the  determination  of  efficiency. 

2 .  Support  Provided 

The  TEA  Division  supports  TRADOC  schools  and  training 
centers  and  provides  input  to  Army  field  units.  The  TEA  Divi¬ 
sion  becomes  involved  in  a  study  by  one  of  two  methods.  The 
proponent  (school,  center,  unit)  submits  a  request  to  TRADOC  or 
directly  to  the  TEA  Division  to  perform  a  study.  If  TRADOC 
determines  the  study  is  of  meric  and  the  TEA  Division  has  re¬ 
sources  available  then  it  will  be  conducted.  A  list  of  major 
study  reports  completed  by  TEA  Division  is  shown  on  Exhibit 
IV-3 . 
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REPORT 

_ 

ABBREV 

NUMBER 

DATE 

Basic  Rifle  Marksmanship  Cost  and 
Training  Effectiveness  Analysis 
(CTEA) 

BRM 

TR-16-77 

SEP  77 

REDEYE  Weapon  Systems  Training 
Effectiveness  Analysis  ( WSTEA) 

RE  WSTEA 

TR  21-77 

NOV  77 

M60A1  Modified  WSTEA 

M6  0A1 

TR  7-78 

JUN  78 

REDEYE  Weapon  Systems  Army 

Training  Study  (ARTS) 

RE  ARTS 

TR  6-78 

OCT  7  8 

TEA  Handbook 

TEA  Hbk 

N/A 

AUG  7  9 

Marksmanship  and  Gunnery  Laser 

Devise/Infantry  Remoted  Target 
System  Training  Developments 

Study  ( TDS ) 

MAG LAD/ 

IRETS 

TEA  1-79 

DEC  7  9 

Multiple  Launch  Rocket  System  CTEA 

MLRS 

TEA  3-80 

JUN  80 

Infantry  Fighting  Vehicle  Initial 

CTEA 

IFV 

TEA  4-80 

MAR  8  0 

Patriot  Air  Defense  System  CTEA 

PATRIOT 

TEA  8-80 

OCT  8  0 

VULCAN  Weapons  System  Training 

Subsystem  Effectiveness  Analysis 
(TSEA) 

VULCAN 

TEA  23-80 

OCT  8  0 

Cavalry  Fighting  Vehicle  Force 

Development  Test  and  Experimenta¬ 
tion  ( FDTE )  Training  Analysis 

CFV 

TEA  31-80 

OCT  8  0 

Armor  Training  Test  Instruments 
and  Selection  Criteria  Evalua¬ 
tion  Study  TEA 

ATS 

TEA  38-80 

JAN  8  1 

Plastic  Ammunition  TDS 

Application:  Military  Opns  in 

PA:  MOUT 

TEA  41-80 

JAN  81 

Urban  Terrain 

Application:  5.56mm  Rifle 

PA:  5.56mm 

TEA  41-80 

JAN  8  1 

Marksmanship  Sustainment 
Firefinder  TEA  operator 

Selection  Criterion 

FIREFINDER 

TEA  4-81 

JAN  8  1 

TDS  for  MOULAGES 

MOULAGES 

TEA  5-81 

FEB  81 

Air  Defense  Accessions  TEA 

ADA  ACC 

TEA  7-81 

MAR  81 

Near-Term  Scout  Helicopter 

Preliminary  CTEA 

NTSH 

TEA  10-81 

APR  81 

CHAPARRAL/REDEYE  TSEA 

CHAP/RE 

TEA  12-81 

JUN  81 

Training  Attrition  Problem, 

Institute  for  Military 

Assistance  TSEA 

TAP IMA 

TEA  13-81 

MAR  81 

Ml  (ABRAMS)  Main  Battle  Tank  TEA 

Ml 

TEA  37-81 

SEP  81 

Air  Defense  Accessions  Update 

ADA  Update 

TEA  40-81 

OCT  81 

TDS  Ml  (ABRAMS)  Tank  Unit-Conduct 
of  Fire  Trainer 

Ml  UCOFT 

TEA  11-82 

MAR  8  2 

Multiple  Int' grated  Laser  Engagement 
System  Air  Ground  Engagement 
Simulation/Air  Defense  CTEA 

MILES  AGES/ 
AD 

TEA  12-82 

MAR  8  2 

ELSAP  2000  Tank  Gunnery  Turret 

Trainer  TDS 

ELSAP 

TEA  13-82 

MAR  8  2 

Ml  ABRAMS  Tank  Driver  Trainer  TDS 

Ml  DVR  TNR 

TEA  15-82 

APR  8  2 

PERSHING  II  TEA 

PERSHING 

TEA  17-82 

MAR  8  2 

UH-60  Flight  Simulator  TDS 

UH60FS 

TDS  19-82 

APR  8  2 

TDS-Bradley  Fighting  Vehicle  Unit- 
Conduct  of  Fire  Trainer 

FV  UCOFT 

TEA  28-82 

MAY  8  2 

Corps  Support  Weapon  System- 
Preliminary  TEA 

CSWS 

TEA  23-82 

JUN  82 

Exhibit  IV-3.  MAJOR  STUDY  REPORTS  COMPLETED 
BY  TEA  DIVISION 
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3 .  Process  Used 

The  TEA  Division  collects  information  through  several  • 

measurement  methods:  standardized  tests  such  as  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB),  Select  Adult  Basic  Learning 
Exam,  Gates  MacGinatie  Reading  Test,  Lynn  Achievement  Motivation 
Scale,  and  Armed  Forces  Qualification  Test  ( AFQT)  Percentile 
Score.  Also,  the  TEA  Division  uses  special  purpose  tests  such 

as  the  Skills  and  Knowledge  (S/K)  tests,  and  performance  pro-  • 

ficiency  tests,  better  known  as  hands-on  tests.  The  above 

listings  are  not  exhaustive  since  additional  sources  of  information 

may  be  included  such  as  demographic  surveys,  force-on-force  sim- 

ulations,  human  factors  analyses,  and  cost  analyses. 

Various  statistical  procedures  are  used  by  the  TEA  Division 
to  evaluate  data  once  it  is  compiled  and  organized.  Several  sta- 
tistical  measures  are  shown  in  Exhibits  IV-4  and  IV-5.  Exactly 
which  procedure  is  used  is  determined  by  the  problem  and  the  nature 
of  the  data.  tM- 


r«L 


»  • 

I 

i 


IV-20 


-  J 


co 
cu 
co  id 
<  o 
w  a 

E-i  u 


z  z 

M  W 

u 
Q  Z 
W  Eh 
co  w 
3  CQ 


CO  W 

a  u 

M  z 
Eh  W 
co  a 

M  W 
Eh  t, 
<  fa 
Eh  m 
CO  Q 


U 

K 

Eh 


W 

Eh 

< 

D 

.J 

<C 

> 

W 


o 

Eh 


ltd 


IV- 21 


CORRELATIONS 

STUDY 

PEARSON 

PRODUCT  MOMENT 

SPEARMAN 

RHO 

REGRESSION 

BRM 

RE  WSTEA 

M60A1 

RE  ARTS 

MAG LAD/ I RETS 

PATRIOT 

VULCAN 

CFV 

ATS 

ADA  ACC 
NTSH 
CHAP/RE 
TAP IMA 
Ml 

PERSHING 
Ml  UCOFT 
FV  UCOFT 


This  was  discriminant  analysis 


Exhibit  IV- 5.  STATISTICS  USED  IN  STUDIES  TO 
DETERMINE  EXTENT  TWO  OR  MORE  VARIABLES  ARE  RELATED 
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REFERENCE  SOURCES 


TASK  1 


A .  DOCUMENTS 


Equipment  Maintenance:  Maintenance  Management,  Air  Force 

Regulation  (AFR)  66-1,  Headquarters,  U.S.  Air  Force,  2  July 
1980. 

Equipment  Maintenance:  Maintenance  Management  Information  and 
Control  System  (MMICS),  Maintenance  Personnel  and  Training 
Management  Users  Manual,  AFM  66-278,  Headquarters,  U.S.  Air 
Force,  1  February  1979. 

Equipment  Maintenance;  Maintenance  Management  Information  and 
Control  System  (MMICS)  Guide  for  Maintenance  Managers,  Air 
Force  Pamphlet  (AFP )  66-10 ,  headquarters ,  U.S.  Air  Force , 

1  April  1980. 

Evaluating  the  Effectiveness  of  Maintenance  Training  by  Using 
Currently  Available  Maintenance  Data,  IDA  Paper  P-1574, 
Institute  for  Defense  Analyses,  August  1981. 

The  Performance  of  Maintenance  Technicians  on  the  Job,  IDA  Paper 
P-1597,  Institute  for  Defense  Analyses,  August  1981. 

Report  on  Individual  Skill  Training — Maintenance  Training  in 
the  Department  of  Defense,  Office  of  the  Assistant  Secretary 
of  Defense  (Manpower,  Reserve  Affairs  and  Logistics),  May 
1982. 


B.  INTERVIEWS 


Name 


Organization  Telephone  ? 


HQ  Air  Force  Staff;  Washington  D.C. 


(AV  22X-XXXX)  or 
(202)  69X-XXXX 


LtCol  Larry  Matthews 
Capt  Freddie  Graham 
Capt  Alfred  Rodriguez 


AF/LEYM  7-1431 
AF/LEYM  7-1493 
AF/MPP  5-7321 


TAC  HQ;  Langley  AFB ,  VA 


(AV  432-XXXX)  or 
(804)  7  64-XXXX 


-*J 


LtCol  William  James 


TAC/LGQT  3688/2102 


Organization 


1st  Tactical  Fighter  Wing;  Langley  AFB,  VA 


LtCol  George  Barr 
MSgt  Fred  Woodard 
TSgt  James  Brown 
MSgt  Roger  Hardwick 
Maj  Ted  Schramm 
SMSgt  Wilson  Davis 
MSgt  Thcmas  Hicks 
MSgt  Ralph  Roberts 
Sgt  Barker 

FTP  201:  Langley  AFB,  VA 


MAT 

MAT 

MAT 

Analysis 

Quality  Assurance 
Quality  Assurance 
Quality  Assurance 
Programs  &  Mobil¬ 
ity  Manpower 


Telephone  # 

(AV  432-XXXX)  or 
(804)  764-XXXX 

2280/3175 

3175 

3610/4390 

7085/7087 


SMSgt  Svabek 

405th  Tactical  Training  Wing:  Luke  AFB,  AZ 


SMSgt  Warren  Smith  MAT 

58th  Tactical  Training  Wing:  Luke  AFB,  AZ 


MSgt  Fancher  MAT 

35th  Tactical  Fighter  Wing:  George  AFB,  CA 


SMSgt  James  Paxton  MAT 

37th  Tactical  Fighter  Wing:  George  AFB,  CA 


LtCol  Fred  Wilson 
SSgt  Randolph  Herrera 
TSgt  Paul  Jones 
TSgt  Ervin  Holt 


(AV  432-XXXX)  or 
(804)  764-XXXX 


(AV  853-XXXX)  or 
(602)  856-XXXX 


(AV  853-XXXX)  or 
(602)  856-XXXX 


(AV  353-XXXX)  or 
(714)  269-XXXX 


(AV  353-XXXX)  or 
(714)  269-XXXX 


Regulation  (AFR)  66-1,  Heaoquarters ,  U.S.  Air  Force,  2  July 
1980. 


Equipment  Maintenance:  Maintenance  Management  Information  and 
Control  System  (MMICS ) ,  Maintenance  Personnel  and  Training  Man¬ 
agement  Users  Manual,  AFM  66-278,  Headquarters,  U.S.  Air  Force, 

1  February  1979. 

Equipment  Maintenance;  Maintenance  Management  Information  and 
Control  System  (MMICS)  Guide  for  Maintenance  Managers,  Air 
Force  Pamphlet  (AFP)  66-10,  Headquarters,  U.S.  Air  Force, 

1  Ap>ril  1980. 

Work  Unit  Code  Manual;  F-16A/B  Aircraft,  Air  Force  Technical 
Manual  T.O.  IF-16A-06,  General  Dynamics  Corp.,  1  February'  1980 
with  7  changes. 

Course  Training  Standards  and  Plans  of  Instruction  for  A1SC  326X6, 
326X7,  326X8,  423XU,  423X2,  and  426X2;  Headquarters  Air  Training 
Command,  1979-1 9a2. 


B.  INTERVIEWS 


Name 

Organization 

Telephone  * 

(AV  22X-XXX)  or 
(202)  69X-XXX 

HQ  Air  Force  Staff:  Washington  D.C* 

LtCol  Larry  Matthews 

AF/LEYM 

7-1431 

Capt  Alfred  Rodriguez 

AF/MPP 

5-7321 

TAC  HQ:  Langley  AFB,  VA 

(AV  432-XXXX)  or 
(804)  764-XXXX 

LtCol  William  James 

TAC/LGQT 

2-3688 

LtCol  Ronald  Clarke 

TAC/LGY 

2-3093 

SMSgt  Spreadbury 

TAC/LGQP 

2-4465 

56th  Tactical  Training  Wing: 

MacDill  AFB,  FL 

(AV  968-XXXX)  or 

(813)  830-XXXX 

MSgt  Warn  Taylor 

MAT 

3212 

Sgt  Stetler 

Analysis 

4708 

388th  Tactical  lighter  Wing: 

Hill  AFB,  UT 

(AV  456-XXXX)  or 

(b01)  777-XXXX 

SMSgt  Robert  Bassett 

MAT 

3835 

Sgt  Childers 

Analysis 

3529 
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B.  INTERVIEWS  (Cont'd) 
Name 


Organization 


Telephone  jf 


474th  Tactical  Fighter  Wing:  Nellis  AFB,  NV 

Maj  David  Crews  MAT 

MSgt  Hobbs  Analysis 

Institute  for  Defense  Analyses;  Alexandria,  VA 

Dr.  Jesse  Orlansky 

Honeywell  Inc.;  West  Covina,  CA 

Dr.  Ruth  Weinclaw 


! 

I 


(AV  682-XXXX)  or 
(702)  643-XXXX 
2213 
2208 


(703)  845-2293 


(213)  331-0011 


TASK  3 


A.  DOCUMENTS 


The  Army  Maintenance  Management  System,  TM  38-750, 
Headquarters,  Department  of  the  Army,  May  1981. 

Evaluating  the  Effectiveness  of  Maintenance  Training  by 

Using  Currently  Available  Maintenance  Data,  IDA  Paper 
P-1574,  Institute  for  Defense  Analyses,  August  1981. 

Maintenance  of  Supplies  and  Equipment;  The  Army  Mainte¬ 
nance  Management  System  (TAMMS)  Reports  and  Summaries 
Catalog ,  DARCOM-P  750-15,  Headquarters,  DARCOM, 

August  1981. 

Maintenance  Performance  System:  Guide  for  Individual 

Technical  Training  in  Direct  Support  Units,  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social 
Sciences,  January  1981. 

Maintenance  Performance  System:  Operator's  Manual,  U.S. 

Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences,  January  1981. 

Maintenance  Performance  System:  Users  Reference  Manual, 
U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  January  1981. 

The  Performance  of  Maintenance  Technicians  on  the  Job,  IDA 
Paper  P-1597,  Institute  for  Defense  Analyses,  August 
1981. 


Report  on  Individual  Skill  Training — Maintenance  Training 
in  the  Department  of  Defense,  Office  of  the  Assistant 
Secretary  of  Defense  (Manpower,  Reserve  Affairs,  and 
Logistics),  May  1982. 

TRADOC  Training  Effectiveness  Handbook  (First  Draft),  U.S. 
Army  TRADOC  Systems  Analysis  Activity,  Undated. 

Training  Effectiveness  Analysis,  A  Process  in  Evolution, 
U.S.  Army  TRADOC  Systems  Analysis  Activity,  October 
1982. 

TRASANA  Report  TEA  37-81,  Ml  (ABRAMS)  Main  Battle  Tank, 

U.S.  Army  TRADOC  Systems  Analysis  Activity,  September 
1981. 
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INTERVIEWS 


Name 

Organization 

Telephone  # 

ARI:  Alexandria,  VA 

(703)  274-XXXX 

Mr.  Doug  Bobko 

ARI 

8694 

Mr.  Mike  Drillings 

ARI 

8694 

Mr.  John  Hayes 

ARI 

8694 

4th  Inf.  Div  (M):  Ft. 

Carson,  CO 

(AV  691-XXXX)  or 

(303)  579-XXXX 

CPT  Robert  Coffman 

Division  G-3 

2938 

Mrs.  Chrol  Koscove 

Division  G-3 

2938 

MAJ  William  Parker 

Division  G-4 

3068 

CPT  Henry  Brown 

Division  G-4 

3068 

MAJ  Daniel  Speck 

704th  Maint . 

Bn. 

2018 

SGT  Giergic 

704th  Maint. 

Bn. 

2018 

SPC  Byassee 

704th  Maint. 

Bn. 

3107 

CPT  Wansersvi 

logistics  School 

5558 

5th  Inf.  Div  (M):  Ft. 

Polk,  LA 

(AV  833-XXXX)  or 

(318)  535-XXXX 

CPT  Peterson 

705th  Maint. 

Bn. 

4180 

SGT  Schaefer 

705th  Maint. 

Bn. 

4180 

Mr.  Sopo 

Logistics  School 

6476 

TRASANA,  White  Sands  Missile  Range,  NM 

(AV  258-XXXX)  or 

(505)  678-XXXX 

Mr.  Lee  Paris 

TRASANA  TEA  Div. 

5915 

Detroit  Diesel  Allison, 

Indianapolis,  IN 

Mr.  Ron  Eber 

Ptein  Office 

(317)  242-3253 

Mr.  Larry  Ritchey 

Main  Office 

(317)  242-6112 

Mr.  Bob  Pejeau 

D.C.  Office 

(202)  775-5058 

APPENDIX  B 

SUMMARY  OUTPUTS  FROM  METHODOLOGICAL  DATABASES 


1  QA  DATA:  SUMMARY  RESULTS 


THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  I  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SXILL  LEVEL  1  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  i  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  1  4  FEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  14  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  7  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  CJT  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  41  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  QA  EXAM 

THERE  WERE  28  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  35  PEOFLE  IN  SKILL  LEVEL  3  WHO  WERE  OJT  AND  FA33ED  THEIR  GA  EXAM 

THERE  WERE  52  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  21  FEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  TRAINED  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  29  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  112  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  QA  EXAM 
THERE  WERE  42  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  TRAINED  AND  FA35ED  THEIR  QA  EXAM 
THERE  WERE  153  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  6  PEGPLE  IN  SKILL  LEVEL  7  WHC  WERE  UNTRAINED  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  13  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  22  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  9  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  QA  EXAM 
THERE  WERE  25  PEOPLE  IN  SKILL  LEVEL  7  WHG  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  32  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  2  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  UNTRAINED  AND  FAS3ED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SXILL  LEVEL  9  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  OJT  AND  PASSED  THEIR  OA  EXAM 
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37  QA  DATA  SUMMARY  RESULTS 


THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  i  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  QA  EXAM 
THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  OJT  AND  FAILED  THEIR  OA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  UNTRAINED  AND  FA33ED  THEIR  QA  EXAM 
THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  ?  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AMD  FAILED  THEIR  QA  EXAM 

THERE  WERE  &  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  1  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  OJT  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  27  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  FASSED  THEIR  QA  EXAM 
THERE  WERE  7  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  10  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  OJT  AND  PASSED  THEIR  QA  EXAM 
THERE  WERE  7  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  QA  EXAM 
THERE  WERE  33  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  17  PEOPLE  IN  SXILL  LEVEL  S  WHO  WERE  OJT  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  31  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  QA  EXAM 

THERE  WERE  42  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  TRAINED  AND  PASSED  THEIR  OA  EXAM 

THERE  WERE  24  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  3  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  OJT  AMD  FAILED  THEIR  QA  EXAM 

THERE  WERE  3  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  QA  EXAM 

THERE  WERE  4  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  TRAINED  AND  PASSED  THEIR  OA  EXAM 

THERE  WERE  8  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  OJT  AND  PASSED  THEIR  OA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  C  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  OJT  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  UNTRAINED  AND  FASSED  THEIR  QA  EXAM 

THERE  WERE  0  PEOPLE  IN  SXILL  LEVEL  7  WHO  WERE  TRAINED  AND  PASSED  THEIR  QA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  OJT  AND  PASSED  THEIR  QA  EXAM 
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405  QA  DATA  SUMMARY  RESULTS 


THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  UNTRAINED  AND  FAILED  TKEIR  GA  EXAM 
THERE  WERE  0  PEOFLE  IN  SKILL  LEVEL  1  WHO  WERE  TRAINED  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  OJT  AND  FAILED  THEIR  QA  EXAM 
THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  i  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  1  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  QA  EXAM 
THERE  WERE  13  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 
THERE  WERE  3  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  3  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  VO  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  4?  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  35  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  QA  EXAM 

THERE  WERE  12?  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  217  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  29  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  186  PEOPLE  IN  SKILL  LEVEL  3  WHO  WERE  TRAINED  AND  PASSED  THEIR  QA  EXAM 
THERE  WERE  281  PEOPLE  IN  SKILL  LEVEL  5  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 
THERE  WERE  8  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  2S  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  TRAINED  AND  FAILED  THEIR  QA  EXAM 
THERE  WERE  V  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 
THERE  WERE  11  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  UNTRAINED  AND  PASSED  THEIR  QA  EXAM 

THERE  WERE  51  PEOPLE  IN  SKILL  LEVEL  7  WHO  WERE  TRAINED  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  7?  PEOPLE  IN  SKILL  LEVEL  7  WHG  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  UNTRAINED  AND  FAILED  THEIR  QA  EXAM 
THERE  WERE  1  PEOPLE  IN  SKILL  LEVEL  9  WHG  WERE  TRAINED  AND  FAILED  TKEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  OJT  AND  FAILED  THEIR  GA  EXAM 

THERE  WERE  0  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  UNTRAINED  AND  FASSED  THEIR  GA  EXAM 
THERE  WERE  3  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  TRAINED  AND  FASSED  THEIR  QA  EXAM 

THERE  WERE  1  PEOPLE  IN  SKILL  LEVEL  9  WHO  WERE  OJT  AND  PASSED  THEIR  GA  EXAM 
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SUMMARY  JViTri  FROM  ETHf.  T<\  RFTORTf: 


STATISTICS  FROM  THE  I/TH  TFW 


TOTAL  «  OF  COMPARISONS  WITHIN  THIS  Kir,=7V 

BET  TEH  TRAINED  WCTRS  PERFORMED  FASTER  58.227848iX  OF  THE  TIKE  FOR  ALL  OBSERVATIONS 
m.WJL  TRAINED  PERCENTAGE  Of  BETTER-TRAINED  NKCTRS^b\*?36/092 
AVERAGE  TRAINED  PERCENTAGE  OF  LESS-TRAINED  NKCTRS‘I4.18*67Jtt 
AVERAGE  DIFFERENCE  IN  TRAINlNG=21.3*)S/V/5>: 

BETTER  TRAINED  SftCTRS  RfckfORnlD  HORN)  ON  THE  AVERAGE*  IN  W.7313675X  OF  THE  TIKE  IT  TOON  LESS-TRAINED  UKCTRS 


ft 


■ 


P 


f 
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SUMMARY  DATA  FROM  WUC  TA  REPORTS 


STATISTICS  FROM  THE  1ST  TFV 

n  • 

"TOTAL  MUM8ER  OF  COMPARISONS  WITHIN  THIS  WING* 17 

BETTER  TRAINED  WKCTRS  PERFORMED  FASTER  51.82351951  Of  THE  TIME  FOR  ALL  OBSERVATIONS  '  V 

AVERAGE  TRAINED  PERCENTAGE  OF  IETTER-TRAINED  WKCTRS  =44.2352941%  .‘X 

•;  AVERAGE  TRAINED  PERCENTAGE  OF  LESS-TRAINED  WKCTRS=25 . 1744704% 

.AVERAGE  DIFFERENCE  IN  TRAININGS  .  0588235% 

SETTER  TRAINED  WKCTRS  PERFORMED  WORK,  ON  THE  AVERAGE,  IN  95.4944159%  OF  THE  TIME  IT  TOOK  LESS-TRAINED  WKCTRS  ’• 

STATISTICS  FROM  THE  405  TTW 

TOTAL  NUMBER  OF  COMPARISONS  WITHIN  THIS  W!NG=119 
"■BETTER  TRAINED  WKCTRS  PERFORMED  FASTER  54.302521%  OF  THE  TIME  FOB  ALL  OBSERVATIONS 
AVERACE  TRAINED  PERCENTAGE  OF  BETTER-TRAINED  WKCTRS=75 . 57 1 4284% 

•  AVERAGE  TRAINED  PERCENTAGE  OF  LESS-TRAINED  WKCTRS*54 . 4349748% 

'  AVERAGE  DIFFERENCE  IN  TRAIN  I HG  ^2  1 . 1344538% 

SETTER  TRAINED  WKCTRS  PERFORMED  WORK,  ON  THE  AVERAGE,  IN  100.518445%  OF  THE  TIME  IT  TOOK  LESS-TRAINED  WKCTRS 

I  * 

STATISTICS  FROM  INTER-WING  COMPARISONS 

•DTAL  NUMBER  OF  INTER-WING  COMPARISONS* 4 2 

IS  COMPARING  THE  1ST  AND  THE  405TH  TOGETHER,  THE  1ST  TFW  HAD  WKCTRS  THAT  PERFORMED  FASTER  AND  WERE  BETTER  TRAINED  24.1935414%  OF  THE  TIM'  \ 

X  COMPARING  THE  1ST  AND  THE  405TH  TOGETHER,  THE  405TH  HAD  WKCTRS  THAT  PERFORMED  FASTER  AND  WERE  BETTER  TRAINED  41.3870948%  OF  THE  TIME, 
■AVERAGE  TRAINED  PERCENTAGE  OF  BETTER-TRAINED  WKCTRS!  IN  INTER-V.NG  COMPARISON)  =71 .80445  1  7%  Jji' 

“AVERAGE  TRAINED  FERCENTACE  OF  LESS-TRAINED  WKCTRS ( IN  INTER-WINC  COMPARISON)* 40. 7419355% 

AVERAGE  DIFFERENCE  IN  TRAININGS  1 . 0445  1  42% 

SETTER  TRAINED  VKCTF.S  PERFORMED  WORK,  OH  THE  AVERAGE,  IN  94  3555434%  OF  THE  TIME  IT  TOOK  LESS-TRAINED  WKCTRS 


OVERALL  STATISTICS.  ALL  COMPARISONS 

■I 

'  TOTAL  NUMBER  OE  CONPARISONS=198 
BETTER  TRAINED  WKCTRS  PERFORMED  FASTER  41.4141414%  OF  THE  TIME 
AVERAGE  TRAINED  PERCENTAGE  OF  BETTER-TRAINED  WKCTHS=73 . 5909091% 

AVERACE  TRAINED  PERCENTAGE  OF  LESS-TRAINED  WKCTR5H7  4343437% 

AVERAGE  DIFFERENCE  IN  TRAININGS . 9545455% 

BETTER  TRAINED  WKCTRS  PERFORMED  WORK,  ON  THE  AVERAGE,  IN  98.8010181%  OF  THE  TIME  IT  TOOK  LESS  TRAINED  WKCTRS 
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This  Appendix  contains  the  following  information: 

•  Information  on  SAMTs  received  from  Honeywell, 

•  Analysis  of  variance  technique,  and 

•  Productivity  and  frequency  calculations. 


INFORMATION  ON  SAMTs 
RECEIVED  FROM  HONEYWELL 
(Included  in  the  handwritten 
provided  to  MCR) 
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ANALYSIS  OF  VARIANCE  (ANOVA)  TECHNIQUE 


Step  1 : 

Assign  all  work  center  observations  (training/frequency/ 
productivity  combinations)  to  a  "cell"  in  an  ANOVA  matrix.  Train¬ 
ing  and  frequency  are  the  independent  varibles.  This  examination 
attempts  to  explain  the  statistical  weight  of  their  impact  on  the 
dependent  variable  (productivity) . 


'^\FREQUENCY 

TRAINING''^ 

LOW 

MEDIUM 

!  ROW 

|  (TRAINING) 
HIGH  1  AVERAGE 

LOW 

— 

MEDIUM 

i 

HIGH 

i 

COLUMN 

(FREQUENCY) 

AVERAGE 

_ 

TOTAL 

AVERAGE 

Based  on  an  analysis  of  the  spread  of  work  center 
observations  included  in  our  analysis,  the  following  defini¬ 
tions  were  adopted  for  the  different  training/frequency  cat¬ 
egories  : 

LOW  TRAINING  —  Work  center  with  <_  20%  of  their  personnel 

trained  in  the  appropriate  skill; 

MEDIUM  TRAINING  --  Work  center  with  >  20%  but  <  60%  of  their 

personnel  trained  in  the  appropriate  skill; 
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HIGH  TRAINING 


—  Work  center  with  >  60%  of  their  personnel 
trained  in  the  appropriate  skill; 

LOW  FREQUENCY  —  Work  center  with  an  action  frequency  of 

_<  1  (less  than/equal  1  action/worker  in 
the  examination  period) ; 

MEDIUM  FREQUENCY  —  Work  center  with  an  action  frequency  >  1 

but  _<  2  (greater  than  1  but  less  than/ equal 
to  2  actions/worker  in  the  examination 
period) . 

HIGH  FREQUENCY  —  Work  center  with  an  action  frequency  >  2 

(greater  than  2  actions/ worker  in  the  ex¬ 
amination  period) . 


STEP  2; 

Once  all  work  center  observations  are  classified  in  the 
appropriate  cell  in  an  ANOVA  matrix,  a  "cell  average"  produc¬ 
tivity  figure  is  computed  in  the  following  manner: 

CELL  AVERAGE  =Z^All  Productivity  Numbers  for  Work  Centers  in  the  Cell 

Number  of  Work  Centers  in  the  Cell 


Similarly,  a  "Row/ Column  average"  productivity  figure  is  computed 
for  each  row/column  in  the  ANOVA  table.  Row  average  productivity 
figures  are  computed  in  the  following  manner: 

ROW  AVERAGE  =^Cell  Average  Productivity  Figures  Within  a  Row 

Number  or  Cells  m  a  Row 

This  average,  in  our  analysis,  provides  a  measure  of  the  impact 
of  a  specific  level  of  training  across  different  frequency  levels. 
Analogously,  column  average  productivity  figures  are  computed  as 
follows : 

COLUMN  AVERAGE  =£cell  Average  Productivity  Figures  Within  a  Column 

Number  of  Cells  in  a  Column 


This  average  provides  a  measure  of  the  impact  of  a  specific 
level  of  frequency  across  different  training  levels. 

In  addition,  a  "row/column  total"  productivity  figure  is 
computed  for  each  row/column  in  the  ANOVA  matrix.  These  totals 
are  simply  defined: 

ROW/COLUMN  TOTAL  =  All  Cell  Average  Productivity  Figures  in  Each 

Row/ Column . 

Finally,  an  "overall  average"  productivity  figure  is  computed 
in  the  following  manner: 

OVERALL  AVERAGE  =]^A11  Cell  Average  Productivity  Figures 

Number  of  Cells 


This  number  provides  a  measure  of  the  average  productivity 
observed  in  our  sample. 

As  with  the  row/column  computations,  an  overall  total 
productivity  figure  is  also  computed.  This  total  is: 

OVERALL  TOTAL  =  All  Cell  Average  Productivity  Figures. 

STEP  3: 

Once  all  averages  and  totals  have  been  computed,  variation 
figures  must  be  computed  for  the  analysis.  The  following  vari¬ 
ation  figures  (provided  with  their  mathematical  definitions)  are 
computed : 

2  2 

V ,  Total  Variation  =  £(Cell  Average  Productivity  Figures)  -  _ _ (Overall  Total)  _ 

(Number  of  Rows) (Number  of  Columns) 

2  2 

VT,  Training  Variation  =  E(Row  Total  Average  Productivity  Figures)  -  ^  _ (Overall  Total) _ _ 

(Number  of  Columns)  (Number  of  Rows) (Number  of  Columns) 

Vp,  Frequency  Variation  =  E(Column  Total  Average  Productivity  Figures)^  -  _ (Overall  Total ) 2 _ 

(.-lumber  of  Rows)  (Number  of  Rows)  (Number  of  Columns) 

V£,  brror  Variation  =  V  (Total  Variation)  -  V^  (Training  Variation)  -  Vp  (Frequency  Variation) 


Oil 


These  statistics  describe  the  variation  of  the  observed 
work  center  productivity  figures  around  an  overall  mean  produc¬ 
tivity  value.  The  formats  are  standard  statistical  variation 
formats,  and  so  no  explanation  of  their  structure  will  be  offered 
here. 

In  our  analysis  of  the  impact  of  training  and  frequency  on 
productivity,  these  variation  figures  are  crucial.  If  it  can 
be  shown  that  the  variation  attributed  to  f requency/training 
is  larger  than  the  error  variation  by  a  certain  amount,  then 
we  can  conclude  that  those  factors  do  have  a  positive  impact 
on  productivity. 

STEP  4 

Calculate  "mean  square"  values  for  the  specific  variations 
computed  in  Step  3.  This  is  a  simple  procedure  where  VT,  Vf,, 
and  VE  are  divided  by  their  associated  "degrees  of  freedom"  to 
come  up  with  a  mean  square  value: 

MS^,  (Mean  Square  Training)  =  VT 

D  F . 

T  (Degrees  of  Freedom  for  Training  Variable) 

MSp  (Mean  Square  Frequency) =  VF 

D  F 

F  (Degrees  of  Freedom  for  Frequency  Variable) 

MS^  (Mean  Square  Error)  =  VE 

D  F 

' E  (Degrees  of  Freedom  for  Error  Variable) 


STEP  5 


Use  the  "mean  square"  values  calculated  in  Step  4  to 
calculate  an  "F-Ratio"  for  both  training  and  frequency. 

The  F-Ratio  is  a  statistic  that  serves  to  measure  the  explan¬ 
atory  power  of  an  independent  variable  on  a  dependent  variable. 

The  ratio  is  computed  in  the  following  form: 

F  .  Mean  Square  Variance  Attributed  to  a  Specific  Independent  Variable 

Mean  Square  Error  Variance 

A  large  F-Ratio  intuitively  says  that  the  independent 
variable  tested  has  a  much  larger  impact  on  the  dependent  vari¬ 
able  than  simple  random  chance,  or,  in  a  statistical  sense,  the 
independent  variable  is  "significant." 

STEP  6 

Once  the  F-Ratios  for  training  and  productivity  have  been 
computed,  the  actual  determination  of  the  significance  of  those 
two  variables  on  productivity  can  be  made.  In  order  to  perform 
this  determination  of  significance,  two  different  hypotheses 
have  to  be  advanced: 

Hq:  Row  (Column)  Average^  =  Row  (Column)  Average.,  =  Row  (Column)  Average-j 

(In  words,  this  hypothesis  states  that  training/frequency  differences  do  not  lead 
to  statistical  differences  in  productivity) . 

and 

Row  (Column)  Average^  /  Row  (Column)  Average2  /  Row  (Column)  Average^ 

(Training/frequency  differences  do  lead  to  statistical  differences  in  productivity) . 
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To  make  a  determination  as  to  which  hypothesis  should  be 
accepted,  the  F-Ratios  for  both  training  and  frequency  computed 
in  Step  5  should  be  compared  against  a  value  of  the  F-Distribu- 
tion  at  a  specific  level  of  confidence.  If  the  F-Ratio  (com¬ 
puted)  >  F-Dist.ribution  (at  a  specific  level  of  confidence), 
then  Hq  is  rejected  and  is  accepted.  In  effect,  this 
conclusion  says  that  the  independent  variable  being  tested 
(training  or  frequency)  does  have  a  statistically  measurable 
impact  on  productivity  at  a  specific  level  of  confidence. 


* 


■ 


PRODUCTIVITY  AND  FREQUENCY  CALCULATIONS 


I 
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*The  total  base-wide  hours  (numerator)  are  divided  by  the  base-wide  frequency 
(denominator)  to  get  the  base-wide  productivity.  However,  in  order  to  get 
a  base  productivity  number  comparable  to  the  individual  work  center  numbers, 
the  result  is  then  divided  by  the  appropriate  number  of  work  centers  at  each 
base  (i.e. ,  2JJ-4  =  58). 
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TRAINING  EFFECTIVENESS  ANALYSIS 
BACKGROUND  MATERIALS 

This  Appendix  provides  additional  information  about 
TRASANA's  TEA  process.  Information  is  presented  in  two 
sections: 

•  the  TRADOC  TEA  Handbook,  and 

•  the  TEA  for  the  Ml  (ABRAMS)  Main  Battle  Tank. 

A.  THE  TRADOC  TRAINING  EFFECTIVENESS  ANALYSIS  (TEA)  HANDBOOK 

The  TRADOC  TEA  Handbook  was  developed  by  TRASANA  to  be  used 
as  a  guidance  document  for  planning,  conducting  and  documenting 
TEA  studies.  The  handbook  is  still  in  draft  and  is  an  evolving 
document.  The  present  description  is  a  summary  of  what  is  con¬ 
tained  in  the  handbook.  The  handbook  covers  three  broad  areas 
which  are  discussed  in  the  subsections  that  follow: 

•  TEA  system  overview, 

•  types  of  TEA,  and 

•  guidance  for  TEA  study  planning. 

1 .  TEA  System  Overview 

The  TEA  system  is  a  management  tool  for  developing  and 
assessing  the  effectiveness  of  training  subsystems  that  are  re¬ 
lated  to  hardware-oriented  systems.  The  TRADOC  TEA  system  can 
be  defined  as  a  series  of  systematic  studies  in  hardware-oriented 
systems  conducted  to  assess  the  impact  of  training  on  system  ef¬ 
fectiveness,  and  to  insure  development  and  implementation  of  cost 
effective  training  subsystems. 


There  are  two  ways  of  classifying  TEAs .  The  primary 
way  is  to  divide  them  according  to  the  phase  of  the  life  cycle 
with  which  a  given  TEA  is  associated.  This  means  on  developing 
hardware-oriented  systems  or  on  fielded  systems  or  on  both.  The 
second  way  has  to  do  with  whether  the  relation  of  cost  and 
effectiveness  is  examined. 


2 .  Types  of  TEA 

The  five  different  types  of  TEAs  are  discussed  below. 

•  Cost  And  Training  Effectiveness  Analysis  (CTEA):  A 
systematic,  continuous  evaluation  process  conducted 
during  the  acquisition  cycle  of  a  hardware-oriented 
system  focusing  on  training  subsystem  development  and 
training  inputs  to  the  Cost  and  Operational  Effective¬ 
ness  Analysis.  This  analysis  addresses  soldier  capa¬ 
bility  to  operate  the  hardware. 

•  Initial  Screening  Training  Effectiveness  Analysis 

(ISTEA):  A  systematic  study  conducted  on  a  fielded 

hardware-oriented  system  to  determine  if  there  is  a 
significant  gap  between  the  Design  Effectiveness  (E  ) 
and  the  Actual  Effectiveness  (E^)  of  the  hardware- 
oriented  system. 

•  Training  Subsystem  Effectiveness  Analysis  (TSEA):  A 
systematic  study  conducted  to  determine  if  the  exis¬ 
tence  of  a  significant  performance  gap  is  partly  or 
entirely  due  to  the  training  subsystem. 

•  Training  Developments  Study  (TDS):  A  systematic 
study  conducted  to  develop  a  fix  for  a  training  sub¬ 
system  found  to  be  deficient  and/or  too  expensive  and 
to  develop  training  devices. 

•  Total  System  Evaluation  (TSE):  A  systematic  evalua¬ 
tion  of  the  hardware,  logistics  support,  and  person¬ 
nel  support  subsystems  of  a  fielded  hardware-oriented 
system.  A  TSE  is  conducted  when  it  has  been  deter¬ 
mined  that  the  training  subsystem  is  neither  the  sole 
nor  primary  cause  of  a  significant  performance  gap. 

Each  of  these  is  described  in  the  subsections  below. 
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a. 


CTEA 


The  CTEA  purposes  are: 

•  comparing  the  cost  and  effectiveness  levels  of  various 
training  subsystem  alternatives; 

•  selecting  the  training  subsystem  alternative  which 
best  supports  and  minimizes  the  cost  associated  with 
the  Army's  training  mission  for  a  specified  hardware 
subsystem,  while  at  the  same  time  insuring  that  per¬ 
formance  standards  of  soldiers  are  at  the  highest 
levels  consistent  with  soldiers'  capabilities;  and 

•  providing  training  subsystem  data  inputs  for  consider¬ 
ation  at  all  decision  points  during  the  acquisition 
cycle. 

A  CTEA  has  the  following  essential  parts: 

•  study  objective, 

•  definition  of  essential  elements  of  analysis  (EEA), 

•  measures  of  training  effectiveness  (MOTE), 

•  data  collection  plan, 

•  data  collection  instruments,  and 

•  data  analysis  plan. 

Thus,  the  CTEA  evaluates  the  cost  and  effective¬ 
ness  of  alternative  training  approaches  as  they  are  being  form¬ 
ulated  to  support  the  developing  total  hardware-oriented  system. 
The  CTEA  is  a  continuous  evaluative  process  that  provides  the 
basis  for  comparing  and  retiring  alternative  training  methods. 


b.  1STEA 


The  ISTEA  purposes  are  to: 

•  determine  if  a  significant  performance  gap  exists 
for  a  fielded  hardware  subsystem,  and 

•  provide  baseline  data  of  soldier  capabilities  to 
support  CTEA  on  newly  developing  hardware  systems. 

An  ISTEA  has  the  same  basic  parts  as  the  CTEA. 
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The  ISTEA  is  primarily  a  quality  control  mechan¬ 
ism  in  the  TRADOC  TEA  system.  The  basic  function  is  to  deter¬ 
mine  if  the  actual  effectiveness  of  a  fielded  hardware  system 
is  significantly  lower  than  its  design  effectiveness.  Further¬ 
more,  the  results  of  an  ISTEA  provide  the  basis  for  any  other 
follow-on  TEAs. 


C.  TSEA 


The  TSEA  purpose  is  to  examine  the  training  sub¬ 
system  in  detail  to  determine  if  an  existing  performance  gap  is 
caused  totally  or  in  part  by  the  training  subsystem.  A  TSEA  has 
the  same  basic  parts  as  the  CTEA. 

The  TSEA  is  usually  conducted  when  an  ISTEA  finds 
a  significant  performance  gap.  The  TSEA  then  determines  if  the 
performance  gap  is  caused  entirely  or  in  part  by  the  training 
subsystem.  A  TSEA  can  be  initiated  in  the  absence  of  an  ISTEA 
when  there  is  other  evidence  of  a  performance  gap  such  as: 

•  widespread  low  Skill  Qualification  Test  (SQT)  scores 
for  system  operators  and  maintainers, 

•  low  annual  service  practice  scores/operational 
readiness  test  scores, 

•  large  numbers  of  weapons  qualification  failures,  or 

•  unsatisfactory  scores  Army-wide  on  Army  Training 
Evaluation  Program  (ARTEP)  tasks  for  units  operating 
a  certain  hardware  subsystem. 

d.  TUS 

The  TDS  purpose  is  to: 

•  develop  a  fix  (change  or  modification)  for  a  train¬ 
ing  subsystem  deficiency, 
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•  develop  a  more  cost  effective  way  to  train, 

•  develop  a  system-related  training  device  for 
either  a  developing  or  a  fielded  system,  and 

•  develop  a  non-system  related  training  device. 

A  TDS  has  the  same  basic  parts  as  the  CTEA. 

The  TDS  is  basically  designed  to  fix  a  training 
subsystem  found  to  be  deficient  and/or  too  expensive. 

e.  TSE 

TSEs  are  conducted  on  fielded  systems  when  it  is 
determined  by  a  TSEA  that  the  training  subsystem  is  neither  the 
sole  or  primary  cause  of  a  performance  gap.  When  the  TSEA  has 
identified  factors  other  than  training  that  directly  or  indi¬ 
rectly  degrade  the  effectiveness  of  the  total  system,  thereby 
causing  a  performance  gap,  a  TSE  is  initiated.  These  factors 
are  typically  elements  of  one  or  more  of  the  following  sub¬ 
systems:  hardware,  logistics  support,  or  personnel  support. 

A  TSE  involves  the  assessment  and  analysis  of  the  foregoing  sub¬ 
systems  to  determine  why  and  how  these  suspect  factors  contri¬ 
bute  to  a  performance  gap.  The  goal  is  to  arrive  at  cost  effec¬ 
tive  solutions  that  will  eliminate  the  performance  gap. 

At  the  present  time  the  TSE  planning  logic  and 
methodology  are  still  being  developed.  Thus  the  TSE  description 
is  not  completed  in  the  TEA  handbook. 

3 .  Guidance  for  TEA  Study  Planning 

This  section  is  a  discussion  of  each  of  the  elements 
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or  steps  that  comprise  a  TEA,  regardless  of  type.  These  steps 
are: 

•  study  design, 

•  data  collection, 

•  data  analysis,  and 

•  results  or  data  interpretation. 

a .  Study  Design 

Study  design  normally  consists  of  the  following 
sequence  of  events  before  arriving  at  the  determination  of  how 
the  study  will  be  performed: 

•  state  the  study  question, 

•  state  the  EEA, 

•  determine  the  sample, 

•  determine  the  treatment, 

•  decide  what  statistical  treatment  is  to  be  used,  and 

•  determine  the  research  design. 

The  study  question  is  the  overall  objective  of 
the  study.  The  EEA  are  the  clear  questions  that  must  be  an¬ 
swered  to  meet  the  study  objective.  They  should  express  a  re¬ 
lationship  between  two  observables  (variables)  that  lend  them¬ 
selves  to  some  form  of  measurement.  The  sample  is  a  subset  of 
the  total  population  which  is  representative,  in  a  statistical 
fashion,  of  the  population.  The  treatment  is  the  type  of  instru 
ment  used  to  measure  some  variable  (e.g.,  SUT  scores).  The 
statistical  treatment  will  depend  upon  the  question  to  be 


answered  and  the  nature  of  the  data.  The  research  design  is 
determined  by  whether  there  is  a  control  group  and  whether  a 
pre-test  and  a  post-test  are  used  in  developing  data. 


b.  Data  Collection 


Data  collection  is  not  governed  by  any  specific 
set  of  rules  other  than  the  advice  to  collect  all  that  is  needed 
and  use  all  valid  data.  Data  collection  can  be  improved  by  us¬ 
ing  experienced  persons  and  by  examining  previous  studies.  The 
only  rules  that  are  rigorously  applied  are  to  insure  that: 

•  all  necessary  data  is  collected, 

•  collection  forms  are  usable,  and 

•  the  data  can  be  easily  coded  for  computer  use. 

I  c.  Data  Analysis 

Data  analysis  varies  from  one  study  to  another. 
The  planning  and  control  of  studies,  however,  will  necessitate 
|  the  following: 

•  data  collation,  or  how  to  aggregate  the  information  in 
a  useful  manner; 

•  check  of  data  characteristics,  which  requires  an  under¬ 
standing  of  the  descriptive  statistics  associated  with 
the  data ; 

•  selection  of  appropriate  statistics,  depending  upon 

the  data  characteristics,  research  design  and  other 
study  characteristics; 

•  selection  of  computer  program/coding;  and 

•  processing  of  the  data. 

Once  the  data  analysis  is  completed,  results  can  be  developed. 

I 
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d.  Data  Interpretation 

Provided  that  a  good  design  is  accomplished,  good 
quality  control  of  data  collection  is  provided  and  use  of  ap¬ 
propriate  statistics  is  completed,  then  answers  can  be  provided 
to  the  EEA.  Exhibit  D-l  shows  the  TEA  planning  process. 

B.  THE  TEA  FOR  THE  Ml  (ABRAMS )  MAIN  BATTLE  TANK 

MCR  requested  the  Ml  TEA  from  TRASANA  because  it  is  the 
Army's  largest  newly  fielded  weapon  system  and  because  the  TEA 
addresses  maintenance  MOSs  which  are  the  primary  interest  of  our 
study.  The  report  presents  the  results  of  the  TEA  of  new  equip¬ 
ment  training  to  transition  M60A1  tank  operators  and  maintainers 
to  become  Ml  tank  qualified.  Data  analyzed  included  demographic 
aptitude,  attitude,  and  instruction  scores  which  are  related  to 
pre-tiaining,  course-imbedded,  and  post-training  hands-on  test 
scores.  Major  proficiency  problems  were  discovered  in  the  oper¬ 
ator  and  turret  mechanic  training  programs.  The  only  deficiency 
noted  for  the  tracked  vehicle  mechanic  was  that  of  trouble¬ 
shooting  using  the  test  set.  In  order  to  narrow  the  scope  of 
our  discussion,  which  is  for  illustrative  purposes  only,  the 
part  of  the  TEA  pertaining  to  the  tracked  vehicle  mechanic  is 
addressed  in  the  following  subsections: 

•  objectives  and  EEA, 

•  methodology  and  test  design, 

•  data  collection, 

•  data  analysis,  and 

•  results  or  findings. 
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Exhibit  D-l.  TEA  PLANNING  PROCESS 


The  objectives  of  the  TEA  were  to: 


•  assess  the  effectiveness  of  transition  training  pro¬ 
grams  for  M60A1  personnel  to  become  proficient  Ml 
personnel,  and 

•  project  the  potential  of  these  programs  when  they  are 
fielded  with  the  Ml  tank. 

The  EEA  stated  in  the  study  plan  are  listed  below. 

•  What  are  the  Ml  tank-specific  critical  tasks  for 
maintenance? 

•  What  are  the  pre-training  proficiencies  of  personnel? 

•  What  are  the  tasks  and  levels  of  effort  included  in 
the  training  program? 

•  What  level  of  proficiency  is  shown  by  trainees  during 
the  training  program? 

•  What  is  the  demonstrated  post-training  proficiency  of 
personnel? 

•  What  performance  deficiencies  were  found  in  post¬ 
training  exercises? 

•  What  are  the  soldier  profiles  (personal  character¬ 
istics)  of  trainers,  students,  and  potential  users  of 
the  training  program? 

«>  What  are  the  attitudes  and  perceptions  of  the  stu¬ 
dents/trainers  of  the  training  and  hardware  subsystem? 


2 .  Methodology/Test  Design 

The  test  design  is  a  pre-test,  training,  and  post-test 
design.  The  pre-test  gathered  demographic,  attitude,  and  pre¬ 
training  aptitude  data.  The  training  phase  provided  the  training 
program  test  results  using  hands-on  tests.  The  post-training 
phase  provided  the  after-training  proficiency  and  attitude  data. 


3. 


Data  Collection 


Data  collection  took  place  in  four  parts  shown  below. 

•  Pre-training — Questionnaires  and  surveys  which  estab¬ 
lished  demographic,  attitude,  motivation,  and  apti¬ 
tude  characteristics  of  each  soldier  to  be  trained 
were  administered  and  hands-on  testing  of  M60A1  skills 
required  for  admission  to  the  transition  training 
program  was  accomplished. 

•  Training--Hands-on  tests  which  assessed  each  stu¬ 
dent's  performance  at  the  end  of  each  major  topic  in 
the  program  were  administered. 

•  Post-training — Hands-on  and  written  tests  of  Ml  skills, 
and  written  attitude  questionnaires  were  completed. 
Additional  post-training  data  was  obtained  through  the 
administration  of  the  post-training  written  test,  ques¬ 
tionnaires  to  determine  attitude  changes,  and  adminis¬ 
tration  of  a  task  criticality  survey. 


4.  Data  Analysis 

The  data  was  collected  using  an  attitude  survey,  moti¬ 
vation  questionnaire,  post-training  hands-on  test,  post-training 
written  test,  and  a  task  criticality  test.  A  statistical  analy¬ 
sis  using  a  multiple  regression  equation  was  attempted;  however, 
due  to  the  number  of  predictive  variables,  no  predictors  of  pro¬ 
ficiency  were  found  for  the  MOS  63E  (tracked  vehicle  mechanic). 
The  TEA  did  find  from  the  hands-on  test  results  that  the  train¬ 
ing  was  effective  for  all  tasks  except  one  (use  of  the  STE/M1 
test  set  for  transmission  troubleshooting). 


5 .  Results  of  Findings 

The  training  program  was  effective  in  all  areas  except 
that  of  troubleshooting  using  the  STE/M1  test  set.  Since  the 
soldiers  were  a  representative  sample  of  the  Army-wide  MOS  63E 
population,  one  would  expect  this  same  result  if  the  same 
program  were  given  to  other  representative  samples. 


D-ll 


